2026-02-22T19:44:40.751 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-22T19:44:40.768 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T19:44:40.851 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972 branch: wip-yuri6-testing-2026-02-22-1629-squid description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '63972' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri6-testing-2026-02-22-1629-squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2026-02-22-1629-squid sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9196 sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuri6-testing-2026-02-22-1629-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 targets: trial042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJFOY7gCzruyFRuLBxtBoDvQgtaYFFQOw5vx3RBuU1mB/Ah/xwnZFQzE4e9LqJjI8XDxcQI6v5hZ7Xf3q0Mv6lg= trial066.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdYkUMhYRZ+Fncb1nPoj2MOd3bMQdmVaMHKfRqH7QQahTfDkUNFuUFQ6ruWGi0llCxilrcV5oFjpJ06/7W/PBc= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: pool RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: journal RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-22_19:35:38 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-22T19:44:40.851 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa; will attempt to use it 2026-02-22T19:44:40.851 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks 2026-02-22T19:44:40.851 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-22T19:44:40.853 INFO:teuthology.task.internal:Checking packages... 2026-02-22T19:44:40.853 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f90f4eaebb28b619cf76e80cbbe997656da69579' 2026-02-22T19:44:40.853 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-22T19:44:40.853 INFO:teuthology.packaging:ref: None 2026-02-22T19:44:40.853 INFO:teuthology.packaging:tag: None 2026-02-22T19:44:40.853 INFO:teuthology.packaging:branch: wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T19:44:40.853 INFO:teuthology.packaging:sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:44:40.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T19:44:40.984 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:44:40.986 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-22T19:44:40.997 INFO:teuthology.task.internal:no buildpackages task found 2026-02-22T19:44:40.997 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-22T19:44:41.014 INFO:teuthology.task.internal:Saving configuration 2026-02-22T19:44:41.027 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-22T19:44:41.041 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-22T19:44:41.103 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial042.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 19:37:54.937854', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeikqcVmaSrSTcNyeeq2WFC0hhAN4wHGOStg/VxlhVT'} 2026-02-22T19:44:41.165 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial066.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 19:37:54.937278', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF+Owb2O2Y4PElAlRSo5QvzTA8q0xH6X7dorusvarOkn'} 2026-02-22T19:44:41.165 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-22T19:44:41.176 INFO:teuthology.task.internal:roles: ubuntu@trial042.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2026-02-22T19:44:41.176 INFO:teuthology.task.internal:roles: ubuntu@trial066.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2026-02-22T19:44:41.177 INFO:teuthology.run_tasks:Running task console_log... 2026-02-22T19:44:41.318 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7e3943866200>, signals=[15]) 2026-02-22T19:44:41.318 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-22T19:44:41.330 INFO:teuthology.task.internal:Opening connections... 2026-02-22T19:44:41.330 DEBUG:teuthology.task.internal:connecting to ubuntu@trial042.front.sepia.ceph.com 2026-02-22T19:44:41.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T19:44:41.393 DEBUG:teuthology.task.internal:connecting to ubuntu@trial066.front.sepia.ceph.com 2026-02-22T19:44:41.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T19:44:41.458 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-22T19:44:41.468 DEBUG:teuthology.orchestra.run.trial042:> uname -m 2026-02-22T19:44:41.472 INFO:teuthology.orchestra.run.trial042.stdout:x86_64 2026-02-22T19:44:41.472 DEBUG:teuthology.orchestra.run.trial042:> cat /etc/os-release 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:NAME="Ubuntu" 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:VERSION_ID="22.04" 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:VERSION_CODENAME=jammy 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:ID=ubuntu 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:ID_LIKE=debian 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-22T19:44:41.517 INFO:teuthology.orchestra.run.trial042.stdout:UBUNTU_CODENAME=jammy 2026-02-22T19:44:41.518 INFO:teuthology.lock.ops:Updating trial042.front.sepia.ceph.com on lock server 2026-02-22T19:44:41.591 DEBUG:teuthology.orchestra.run.trial066:> uname -m 2026-02-22T19:44:41.595 INFO:teuthology.orchestra.run.trial066.stdout:x86_64 2026-02-22T19:44:41.595 DEBUG:teuthology.orchestra.run.trial066:> cat /etc/os-release 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:NAME="Ubuntu" 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:VERSION_ID="22.04" 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:VERSION_CODENAME=jammy 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:ID=ubuntu 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:ID_LIKE=debian 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-22T19:44:41.639 INFO:teuthology.orchestra.run.trial066.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-22T19:44:41.640 INFO:teuthology.orchestra.run.trial066.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-22T19:44:41.640 INFO:teuthology.orchestra.run.trial066.stdout:UBUNTU_CODENAME=jammy 2026-02-22T19:44:41.640 INFO:teuthology.lock.ops:Updating trial066.front.sepia.ceph.com on lock server 2026-02-22T19:44:41.701 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-22T19:44:41.717 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-22T19:44:41.730 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-22T19:44:41.730 DEBUG:teuthology.orchestra.run.trial042:> test '!' -e /home/ubuntu/cephtest 2026-02-22T19:44:41.731 DEBUG:teuthology.orchestra.run.trial066:> test '!' -e /home/ubuntu/cephtest 2026-02-22T19:44:41.735 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-22T19:44:41.749 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-22T19:44:41.749 DEBUG:teuthology.orchestra.run.trial042:> test -z $(ls -A /var/lib/ceph) 2026-02-22T19:44:41.775 DEBUG:teuthology.orchestra.run.trial066:> test -z $(ls -A /var/lib/ceph) 2026-02-22T19:44:41.778 INFO:teuthology.orchestra.run.trial042.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-22T19:44:41.780 INFO:teuthology.orchestra.run.trial066.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-22T19:44:41.780 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-22T19:44:41.927 INFO:teuthology.run_tasks:Running task kernel... 2026-02-22T19:44:41.948 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-22T19:44:41.948 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-22T19:44:41.948 DEBUG:teuthology.orchestra.run.trial042:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T19:44:41.948 DEBUG:teuthology.orchestra.run.trial066:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T19:44:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:44:41.952 DEBUG:teuthology.orchestra.run.trial066:> uname -r 2026-02-22T19:44:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:44:41.952 DEBUG:teuthology.orchestra.run.trial042:> uname -r 2026-02-22T19:44:41.996 INFO:teuthology.orchestra.run.trial066.stdout:5.15.0-170-generic 2026-02-22T19:44:41.996 INFO:teuthology.task.kernel:Running kernel on trial066: 5.15.0-170-generic 2026-02-22T19:44:41.996 DEBUG:teuthology.orchestra.run.trial066:> sudo apt-get clean 2026-02-22T19:44:41.997 INFO:teuthology.orchestra.run.trial042.stdout:5.15.0-170-generic 2026-02-22T19:44:41.997 INFO:teuthology.task.kernel:Running kernel on trial042: 5.15.0-170-generic 2026-02-22T19:44:41.997 DEBUG:teuthology.orchestra.run.trial042:> sudo apt-get clean 2026-02-22T19:44:42.054 DEBUG:teuthology.orchestra.run.trial042:> sudo apt-get update 2026-02-22T19:44:42.055 DEBUG:teuthology.orchestra.run.trial066:> sudo apt-get update 2026-02-22T19:44:42.252 INFO:teuthology.orchestra.run.trial066.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T19:44:42.259 INFO:teuthology.orchestra.run.trial066.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-22T19:44:42.298 INFO:teuthology.orchestra.run.trial066.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-22T19:44:42.314 INFO:teuthology.orchestra.run.trial066.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-22T19:44:42.395 INFO:teuthology.orchestra.run.trial066.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-22T19:44:42.460 INFO:teuthology.orchestra.run.trial066.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-22T19:44:42.463 INFO:teuthology.orchestra.run.trial066.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-22T19:44:42.463 INFO:teuthology.orchestra.run.trial066.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-22T19:44:42.470 INFO:teuthology.orchestra.run.trial066.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-22T19:44:42.472 INFO:teuthology.orchestra.run.trial066.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-22T19:44:42.472 INFO:teuthology.orchestra.run.trial066.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-22T19:44:42.500 INFO:teuthology.orchestra.run.trial066.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-22T19:44:42.503 INFO:teuthology.orchestra.run.trial066.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-22T19:44:42.512 INFO:teuthology.orchestra.run.trial066.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-22T19:44:42.513 INFO:teuthology.orchestra.run.trial066.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-22T19:44:42.513 INFO:teuthology.orchestra.run.trial066.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-22T19:44:42.516 INFO:teuthology.orchestra.run.trial066.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-22T19:44:42.516 INFO:teuthology.orchestra.run.trial066.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-22T19:44:42.533 INFO:teuthology.orchestra.run.trial066.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-22T19:44:42.569 INFO:teuthology.orchestra.run.trial042.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T19:44:42.664 INFO:teuthology.orchestra.run.trial042.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-22T19:44:43.017 INFO:teuthology.orchestra.run.trial042.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-22T19:44:43.172 INFO:teuthology.orchestra.run.trial042.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-22T19:44:43.317 INFO:teuthology.orchestra.run.trial042.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-22T19:44:43.737 INFO:teuthology.orchestra.run.trial042.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-22T19:44:43.750 INFO:teuthology.orchestra.run.trial042.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-22T19:44:43.751 INFO:teuthology.orchestra.run.trial042.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-22T19:44:43.786 INFO:teuthology.orchestra.run.trial042.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-22T19:44:43.795 INFO:teuthology.orchestra.run.trial042.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-22T19:44:43.796 INFO:teuthology.orchestra.run.trial042.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-22T19:44:43.959 INFO:teuthology.orchestra.run.trial042.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-22T19:44:43.987 INFO:teuthology.orchestra.run.trial042.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-22T19:44:44.074 INFO:teuthology.orchestra.run.trial042.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-22T19:44:44.090 INFO:teuthology.orchestra.run.trial042.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-22T19:44:44.090 INFO:teuthology.orchestra.run.trial042.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-22T19:44:44.122 INFO:teuthology.orchestra.run.trial042.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-22T19:44:44.122 INFO:teuthology.orchestra.run.trial042.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-22T19:44:44.280 INFO:teuthology.orchestra.run.trial042.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-22T19:44:46.517 INFO:teuthology.orchestra.run.trial066.stdout:Fetched 22.6 MB in 2s (13.6 MB/s) 2026-02-22T19:44:47.034 INFO:teuthology.orchestra.run.trial066.stdout:Reading package lists... 2026-02-22T19:44:47.047 DEBUG:teuthology.orchestra.run.trial066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-22T19:44:47.080 INFO:teuthology.orchestra.run.trial066.stdout:Reading package lists... 2026-02-22T19:44:47.185 INFO:teuthology.orchestra.run.trial066.stdout:Building dependency tree... 2026-02-22T19:44:47.186 INFO:teuthology.orchestra.run.trial066.stdout:Reading state information... 2026-02-22T19:44:47.271 INFO:teuthology.orchestra.run.trial066.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-22T19:44:47.271 INFO:teuthology.orchestra.run.trial066.stdout:linux-image-generic set to manually installed. 2026-02-22T19:44:47.271 INFO:teuthology.orchestra.run.trial066.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T19:44:47.271 INFO:teuthology.orchestra.run.trial066.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T19:44:47.271 INFO:teuthology.orchestra.run.trial066.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T19:44:47.271 INFO:teuthology.orchestra.run.trial066.stdout: sg3-utils-udev 2026-02-22T19:44:47.271 INFO:teuthology.orchestra.run.trial066.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T19:44:47.290 INFO:teuthology.orchestra.run.trial066.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T19:44:47.292 DEBUG:teuthology.orchestra.run.trial066:> dpkg -s linux-image-generic 2026-02-22T19:44:47.342 INFO:teuthology.orchestra.run.trial066.stdout:Package: linux-image-generic 2026-02-22T19:44:47.342 INFO:teuthology.orchestra.run.trial066.stdout:Status: install ok installed 2026-02-22T19:44:47.342 INFO:teuthology.orchestra.run.trial066.stdout:Priority: optional 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Section: kernel 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Installed-Size: 21 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Maintainer: Ubuntu Kernel Team 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Architecture: amd64 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Source: linux-meta 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Version: 5.15.0.170.159 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Recommends: thermald 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout:Description: Generic Linux kernel image 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout: This package will always depend on the latest generic kernel image 2026-02-22T19:44:47.343 INFO:teuthology.orchestra.run.trial066.stdout: available. 2026-02-22T19:44:47.343 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T19:44:47.343 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2026-02-22T19:44:47.343 DEBUG:teuthology.orchestra.run.trial066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T19:44:47.392 INFO:teuthology.orchestra.run.trial066.stdout:ttyS1 2026-02-22T19:44:47.414 DEBUG:teuthology.parallel:result is None 2026-02-22T19:44:47.684 INFO:teuthology.orchestra.run.trial042.stdout:Fetched 22.6 MB in 3s (7586 kB/s) 2026-02-22T19:44:48.218 INFO:teuthology.orchestra.run.trial042.stdout:Reading package lists... 2026-02-22T19:44:48.231 DEBUG:teuthology.orchestra.run.trial042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-22T19:44:48.264 INFO:teuthology.orchestra.run.trial042.stdout:Reading package lists... 2026-02-22T19:44:48.373 INFO:teuthology.orchestra.run.trial042.stdout:Building dependency tree... 2026-02-22T19:44:48.374 INFO:teuthology.orchestra.run.trial042.stdout:Reading state information... 2026-02-22T19:44:48.462 INFO:teuthology.orchestra.run.trial042.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-22T19:44:48.463 INFO:teuthology.orchestra.run.trial042.stdout:linux-image-generic set to manually installed. 2026-02-22T19:44:48.463 INFO:teuthology.orchestra.run.trial042.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T19:44:48.463 INFO:teuthology.orchestra.run.trial042.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T19:44:48.463 INFO:teuthology.orchestra.run.trial042.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T19:44:48.463 INFO:teuthology.orchestra.run.trial042.stdout: sg3-utils-udev 2026-02-22T19:44:48.463 INFO:teuthology.orchestra.run.trial042.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T19:44:48.483 INFO:teuthology.orchestra.run.trial042.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T19:44:48.484 DEBUG:teuthology.orchestra.run.trial042:> dpkg -s linux-image-generic 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Package: linux-image-generic 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Status: install ok installed 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Priority: optional 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Section: kernel 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Installed-Size: 21 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Maintainer: Ubuntu Kernel Team 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Architecture: amd64 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Source: linux-meta 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Version: 5.15.0.170.159 2026-02-22T19:44:48.535 INFO:teuthology.orchestra.run.trial042.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-22T19:44:48.536 INFO:teuthology.orchestra.run.trial042.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-22T19:44:48.536 INFO:teuthology.orchestra.run.trial042.stdout:Recommends: thermald 2026-02-22T19:44:48.536 INFO:teuthology.orchestra.run.trial042.stdout:Description: Generic Linux kernel image 2026-02-22T19:44:48.536 INFO:teuthology.orchestra.run.trial042.stdout: This package will always depend on the latest generic kernel image 2026-02-22T19:44:48.536 INFO:teuthology.orchestra.run.trial042.stdout: available. 2026-02-22T19:44:48.536 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T19:44:48.536 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2026-02-22T19:44:48.536 DEBUG:teuthology.orchestra.run.trial042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T19:44:48.586 INFO:teuthology.orchestra.run.trial042.stdout:ttyS1 2026-02-22T19:44:48.611 DEBUG:teuthology.parallel:result is None 2026-02-22T19:44:48.611 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-22T19:44:48.623 INFO:teuthology.task.internal:Creating test directory... 2026-02-22T19:44:48.623 DEBUG:teuthology.orchestra.run.trial042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T19:44:48.655 DEBUG:teuthology.orchestra.run.trial066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T19:44:48.659 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-22T19:44:48.670 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-22T19:44:48.684 INFO:teuthology.task.internal:Creating archive directory... 2026-02-22T19:44:48.684 DEBUG:teuthology.orchestra.run.trial042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T19:44:48.703 DEBUG:teuthology.orchestra.run.trial066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T19:44:48.709 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-22T19:44:48.721 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-22T19:44:48.721 DEBUG:teuthology.orchestra.run.trial042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T19:44:48.751 DEBUG:teuthology.orchestra.run.trial066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T19:44:48.758 INFO:teuthology.orchestra.run.trial042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T19:44:48.761 INFO:teuthology.orchestra.run.trial066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T19:44:48.764 INFO:teuthology.orchestra.run.trial042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T19:44:48.767 INFO:teuthology.orchestra.run.trial066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T19:44:48.768 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-22T19:44:48.784 INFO:teuthology.task.internal:Configuring sudo... 2026-02-22T19:44:48.784 DEBUG:teuthology.orchestra.run.trial042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T19:44:48.811 DEBUG:teuthology.orchestra.run.trial066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T19:44:48.820 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-22T19:44:48.832 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-22T19:44:48.832 DEBUG:teuthology.orchestra.run.trial042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T19:44:48.863 DEBUG:teuthology.orchestra.run.trial066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T19:44:48.868 DEBUG:teuthology.orchestra.run.trial042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T19:44:48.913 DEBUG:teuthology.orchestra.run.trial042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T19:44:48.961 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:44:48.961 DEBUG:teuthology.orchestra.run.trial042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T19:44:49.010 DEBUG:teuthology.orchestra.run.trial066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T19:44:49.014 DEBUG:teuthology.orchestra.run.trial066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T19:44:49.064 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:44:49.064 DEBUG:teuthology.orchestra.run.trial066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T19:44:49.118 DEBUG:teuthology.orchestra.run.trial042:> sudo service rsyslog restart 2026-02-22T19:44:49.119 DEBUG:teuthology.orchestra.run.trial066:> sudo service rsyslog restart 2026-02-22T19:44:49.178 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-22T19:44:49.189 INFO:teuthology.task.internal:Starting timer... 2026-02-22T19:44:49.190 INFO:teuthology.run_tasks:Running task pcp... 2026-02-22T19:44:49.207 INFO:teuthology.run_tasks:Running task selinux... 2026-02-22T19:44:49.219 DEBUG:teuthology.task.selinux:Excluding trial042: OS 'ubuntu' does not support SELinux 2026-02-22T19:44:49.219 DEBUG:teuthology.task.selinux:Excluding trial066: OS 'ubuntu' does not support SELinux 2026-02-22T19:44:49.219 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-22T19:44:49.219 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-22T19:44:49.219 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-22T19:44:49.219 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-22T19:44:49.234 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-22T19:44:49.241 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-22T19:44:49.241 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial042.front.sepia.ceph.com,trial066.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-22T19:46:55.637 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial042.front.sepia.ceph.com'), Remote(name='ubuntu@trial066.front.sepia.ceph.com')] 2026-02-22T19:46:55.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial042.front.sepia.ceph.com' 2026-02-22T19:46:55.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T19:46:55.712 DEBUG:teuthology.orchestra.run.trial042:> true 2026-02-22T19:46:55.964 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial042.front.sepia.ceph.com' 2026-02-22T19:46:55.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial066.front.sepia.ceph.com' 2026-02-22T19:46:55.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T19:46:56.066 DEBUG:teuthology.orchestra.run.trial066:> true 2026-02-22T19:46:56.315 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial066.front.sepia.ceph.com' 2026-02-22T19:46:56.315 INFO:teuthology.run_tasks:Running task clock... 2026-02-22T19:46:56.334 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-22T19:46:56.334 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T19:46:56.335 DEBUG:teuthology.orchestra.run.trial042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T19:46:56.337 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T19:46:56.337 DEBUG:teuthology.orchestra.run.trial066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T19:46:56.354 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-22T19:46:56.354 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: Command line: ntpd -gq 2026-02-22T19:46:56.354 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: ---------------------------------------------------- 2026-02-22T19:46:56.354 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: ntp-4 is maintained by Network Time Foundation, 2026-02-22T19:46:56.354 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-22T19:46:56.354 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: corporation. Support and training for ntp-4 are 2026-02-22T19:46:56.354 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: available at https://www.nwtime.org/support 2026-02-22T19:46:56.354 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: ---------------------------------------------------- 2026-02-22T19:46:56.355 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: proto: precision = 0.030 usec (-25) 2026-02-22T19:46:56.355 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: basedate set to 2022-02-04 2026-02-22T19:46:56.355 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: gps base set to 2022-02-06 (week 2196) 2026-02-22T19:46:56.355 INFO:teuthology.orchestra.run.trial042.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T19:46:56.355 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T19:46:56.355 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: restrict ::: KOD does nothing without LIMITED. 2026-02-22T19:46:56.355 INFO:teuthology.orchestra.run.trial042.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-22T19:46:56.355 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: Listen and drop on 0 v6wildcard [::]:123 2026-02-22T19:46:56.356 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-22T19:46:56.356 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: Listen normally on 2 lo 127.0.0.1:123 2026-02-22T19:46:56.356 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: Listen normally on 3 enp1s0f0 10.20.193.42:123 2026-02-22T19:46:56.356 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: Listen normally on 4 lo [::1]:123 2026-02-22T19:46:56.356 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:60aa%2]:123 2026-02-22T19:46:56.356 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:46:56 ntpd[9319]: Listening on routing socket on fd #22 for interface updates 2026-02-22T19:46:56.379 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-22T19:46:56.379 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: Command line: ntpd -gq 2026-02-22T19:46:56.379 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: ---------------------------------------------------- 2026-02-22T19:46:56.379 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: ntp-4 is maintained by Network Time Foundation, 2026-02-22T19:46:56.379 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-22T19:46:56.379 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: corporation. Support and training for ntp-4 are 2026-02-22T19:46:56.379 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: available at https://www.nwtime.org/support 2026-02-22T19:46:56.379 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: ---------------------------------------------------- 2026-02-22T19:46:56.380 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: proto: precision = 0.050 usec (-24) 2026-02-22T19:46:56.380 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: basedate set to 2022-02-04 2026-02-22T19:46:56.380 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: gps base set to 2022-02-06 (week 2196) 2026-02-22T19:46:56.380 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T19:46:56.380 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: restrict ::: KOD does nothing without LIMITED. 2026-02-22T19:46:56.380 INFO:teuthology.orchestra.run.trial066.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T19:46:56.380 INFO:teuthology.orchestra.run.trial066.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-22T19:46:56.380 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: Listen and drop on 0 v6wildcard [::]:123 2026-02-22T19:46:56.380 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-22T19:46:56.381 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: Listen normally on 2 lo 127.0.0.1:123 2026-02-22T19:46:56.381 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: Listen normally on 3 enp1s0f0 10.20.193.66:123 2026-02-22T19:46:56.381 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: Listen normally on 4 lo [::1]:123 2026-02-22T19:46:56.381 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:5f2c%2]:123 2026-02-22T19:46:56.381 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:46:56 ntpd[9444]: Listening on routing socket on fd #22 for interface updates 2026-02-22T19:46:57.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:57.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:57.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:57.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:57 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:58.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:58 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:58.355 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:58 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:58.368 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:58 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:58.368 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:58 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:58.368 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:58 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:58.368 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:58 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:58.379 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:58 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:58.379 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:58 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:58.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:58 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:58.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:58 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:58.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:58 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:58.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:58 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:59.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.355 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.355 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.355 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.379 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.399 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.399 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.399 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:59.400 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:59.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.422 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.422 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:59.422 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:59.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:59.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:46:59 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:59.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:59.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:59.449 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.449 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.449 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:59.449 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:46:59 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:00.355 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:00.355 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:00.355 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:00.355 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:00.358 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:00.358 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:00.358 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:00.358 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:00.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:00.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:00.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:00.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:00 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:00.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:00 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:00.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:00 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:00.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:00 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:00.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:00 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:00.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:00 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:00.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:00 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:01.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.403 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.403 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.403 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:01.403 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:01.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:01.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:01.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:01.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:01 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:01.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:01.447 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:01.449 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:01.449 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:01.449 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:01.449 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:01 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:02.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:02.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:02.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:02.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:02.357 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:02.357 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:02.357 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:02.357 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:02.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:02.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:02.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:02.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:02 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:02.379 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:02 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:02.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:02 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:02.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:02 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:02.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:02 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:02.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:02 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:02.413 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:02 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:03.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:03.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:03.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:03.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:03.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:03.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:03.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:03.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:03.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:03.380 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:03.399 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:03.399 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:03.399 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:03.399 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:03.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:03.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:03.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:03.421 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:03.421 INFO:teuthology.orchestra.run.trial066.stdout:22 Feb 19:47:03 ntpd[9444]: ntpd: time slew -0.000682 s 2026-02-22T19:47:03.421 INFO:teuthology.orchestra.run.trial066.stdout:ntpd: time slew -0.000682s 2026-02-22T19:47:03.422 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-22T19:47:03.422 INFO:teuthology.orchestra.run.trial066.stderr:22 Feb 19:47:03 ntpd[9444]: can't open /var/log/ntpstats/loopstats.20260222: Permission denied 2026-02-22T19:47:03.427 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:03.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:03.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:03.428 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:03 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:03.483 INFO:teuthology.orchestra.run.trial066.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T19:47:03.483 INFO:teuthology.orchestra.run.trial066.stdout:============================================================================== 2026-02-22T19:47:03.483 INFO:teuthology.orchestra.run.trial066.stdout: 172-104-28-175. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:47:03.483 INFO:teuthology.orchestra.run.trial066.stdout: nyc3.us.ntp.li .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:47:03.484 INFO:teuthology.orchestra.run.trial066.stdout: 138.89.14.60 (p .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:47:03.484 INFO:teuthology.orchestra.run.trial066.stdout: LAX.CALTICK.NET .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:47:04.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:04.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:04.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:04.354 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:04.357 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:04.357 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:04.357 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:04.357 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:04.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:47:04.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:47:04.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:47:04.362 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:47:04.362 INFO:teuthology.orchestra.run.trial042.stdout:22 Feb 19:47:04 ntpd[9319]: ntpd: time set -0.056847 s 2026-02-22T19:47:04.362 INFO:teuthology.orchestra.run.trial042.stdout:ntpd: time set -0.056847s 2026-02-22T19:47:04.363 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-22T19:47:04.363 INFO:teuthology.orchestra.run.trial042.stderr:22 Feb 19:47:04 ntpd[9319]: can't open /var/log/ntpstats/loopstats.20260222: Permission denied 2026-02-22T19:47:04.402 INFO:teuthology.orchestra.run.trial042.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T19:47:04.402 INFO:teuthology.orchestra.run.trial042.stdout:============================================================================== 2026-02-22T19:47:04.403 INFO:teuthology.orchestra.run.trial042.stdout: 172-104-28-175. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:47:04.403 INFO:teuthology.orchestra.run.trial042.stdout: as393746.mci.tr .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:47:04.403 INFO:teuthology.orchestra.run.trial042.stdout: 138.89.14.60 (p .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:47:04.403 INFO:teuthology.orchestra.run.trial042.stdout: 73.185.182.209 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:47:04.404 INFO:teuthology.run_tasks:Running task install... 2026-02-22T19:47:04.416 DEBUG:teuthology.task.install:project ceph 2026-02-22T19:47:04.416 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}} 2026-02-22T19:47:04.417 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'} 2026-02-22T19:47:04.417 INFO:teuthology.task.install:Using flavor: default 2026-02-22T19:47:04.426 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-22T19:47:04.426 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2026-02-22T19:47:04.427 DEBUG:teuthology.orchestra.run.trial042:> sudo apt-key list | grep Ceph 2026-02-22T19:47:04.427 DEBUG:teuthology.orchestra.run.trial066:> sudo apt-key list | grep Ceph 2026-02-22T19:47:04.481 INFO:teuthology.orchestra.run.trial066.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-22T19:47:04.497 INFO:teuthology.orchestra.run.trial042.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-22T19:47:04.509 INFO:teuthology.orchestra.run.trial066.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-22T19:47:04.509 INFO:teuthology.orchestra.run.trial066.stdout:uid [ unknown] Ceph.com (release key) 2026-02-22T19:47:04.510 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2026-02-22T19:47:04.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:47:04.587 INFO:teuthology.orchestra.run.trial042.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-22T19:47:04.587 INFO:teuthology.orchestra.run.trial042.stdout:uid [ unknown] Ceph.com (release key) 2026-02-22T19:47:04.587 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2026-02-22T19:47:04.587 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:47:04.685 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default/ 2026-02-22T19:47:04.685 INFO:teuthology.task.install.deb:Package version is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:47:04.707 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default/ 2026-02-22T19:47:04.707 INFO:teuthology.task.install.deb:Package version is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:47:05.025 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:47:05.025 DEBUG:teuthology.orchestra.run.trial066:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-22T19:47:05.035 DEBUG:teuthology.orchestra.run.trial066:> sudo apt-get update 2026-02-22T19:47:05.065 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:47:05.065 DEBUG:teuthology.orchestra.run.trial042:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-22T19:47:05.077 DEBUG:teuthology.orchestra.run.trial042:> sudo apt-get update 2026-02-22T19:47:05.253 INFO:teuthology.orchestra.run.trial042.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T19:47:05.256 INFO:teuthology.orchestra.run.trial042.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-22T19:47:05.265 INFO:teuthology.orchestra.run.trial042.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-22T19:47:05.274 INFO:teuthology.orchestra.run.trial042.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-22T19:47:05.419 INFO:teuthology.orchestra.run.trial066.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy InRelease 2026-02-22T19:47:05.459 INFO:teuthology.orchestra.run.trial042.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy InRelease 2026-02-22T19:47:05.486 INFO:teuthology.orchestra.run.trial066.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release [7662 B] 2026-02-22T19:47:05.522 INFO:teuthology.orchestra.run.trial042.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release [7662 B] 2026-02-22T19:47:05.549 INFO:teuthology.orchestra.run.trial066.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T19:47:05.551 INFO:teuthology.orchestra.run.trial066.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-22T19:47:05.586 INFO:teuthology.orchestra.run.trial042.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-22T19:47:05.617 INFO:teuthology.orchestra.run.trial066.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-22T19:47:05.635 INFO:teuthology.orchestra.run.trial066.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-22T19:47:05.653 INFO:teuthology.orchestra.run.trial042.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-22T19:47:05.727 INFO:teuthology.orchestra.run.trial066.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-22T19:47:05.813 INFO:teuthology.orchestra.run.trial042.stdout:Fetched 25.8 kB in 1s (44.9 kB/s) 2026-02-22T19:47:05.819 INFO:teuthology.orchestra.run.trial066.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-22T19:47:05.960 INFO:teuthology.orchestra.run.trial066.stdout:Fetched 25.8 kB in 1s (33.8 kB/s) 2026-02-22T19:47:06.328 INFO:teuthology.orchestra.run.trial042.stdout:Reading package lists... 2026-02-22T19:47:06.341 DEBUG:teuthology.orchestra.run.trial042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-689-gf90f4eae-1jammy cephadm=19.2.3-689-gf90f4eae-1jammy ceph-mds=19.2.3-689-gf90f4eae-1jammy ceph-mgr=19.2.3-689-gf90f4eae-1jammy ceph-common=19.2.3-689-gf90f4eae-1jammy ceph-fuse=19.2.3-689-gf90f4eae-1jammy ceph-test=19.2.3-689-gf90f4eae-1jammy ceph-volume=19.2.3-689-gf90f4eae-1jammy radosgw=19.2.3-689-gf90f4eae-1jammy python3-rados=19.2.3-689-gf90f4eae-1jammy python3-rgw=19.2.3-689-gf90f4eae-1jammy python3-cephfs=19.2.3-689-gf90f4eae-1jammy python3-rbd=19.2.3-689-gf90f4eae-1jammy libcephfs2=19.2.3-689-gf90f4eae-1jammy libcephfs-dev=19.2.3-689-gf90f4eae-1jammy librados2=19.2.3-689-gf90f4eae-1jammy librbd1=19.2.3-689-gf90f4eae-1jammy rbd-fuse=19.2.3-689-gf90f4eae-1jammy rbd-mirror=19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:47:06.374 INFO:teuthology.orchestra.run.trial042.stdout:Reading package lists... 2026-02-22T19:47:06.481 INFO:teuthology.orchestra.run.trial042.stdout:Building dependency tree... 2026-02-22T19:47:06.481 INFO:teuthology.orchestra.run.trial042.stdout:Reading state information... 2026-02-22T19:47:06.487 INFO:teuthology.orchestra.run.trial066.stdout:Reading package lists... 2026-02-22T19:47:06.501 DEBUG:teuthology.orchestra.run.trial066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-689-gf90f4eae-1jammy cephadm=19.2.3-689-gf90f4eae-1jammy ceph-mds=19.2.3-689-gf90f4eae-1jammy ceph-mgr=19.2.3-689-gf90f4eae-1jammy ceph-common=19.2.3-689-gf90f4eae-1jammy ceph-fuse=19.2.3-689-gf90f4eae-1jammy ceph-test=19.2.3-689-gf90f4eae-1jammy ceph-volume=19.2.3-689-gf90f4eae-1jammy radosgw=19.2.3-689-gf90f4eae-1jammy python3-rados=19.2.3-689-gf90f4eae-1jammy python3-rgw=19.2.3-689-gf90f4eae-1jammy python3-cephfs=19.2.3-689-gf90f4eae-1jammy python3-rbd=19.2.3-689-gf90f4eae-1jammy libcephfs2=19.2.3-689-gf90f4eae-1jammy libcephfs-dev=19.2.3-689-gf90f4eae-1jammy librados2=19.2.3-689-gf90f4eae-1jammy librbd1=19.2.3-689-gf90f4eae-1jammy rbd-fuse=19.2.3-689-gf90f4eae-1jammy rbd-mirror=19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:47:06.533 INFO:teuthology.orchestra.run.trial066.stdout:Reading package lists... 2026-02-22T19:47:06.569 INFO:teuthology.orchestra.run.trial042.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T19:47:06.570 INFO:teuthology.orchestra.run.trial042.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T19:47:06.570 INFO:teuthology.orchestra.run.trial042.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T19:47:06.570 INFO:teuthology.orchestra.run.trial042.stdout: sg3-utils-udev 2026-02-22T19:47:06.570 INFO:teuthology.orchestra.run.trial042.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T19:47:06.570 INFO:teuthology.orchestra.run.trial042.stdout:The following additional packages will be installed: 2026-02-22T19:47:06.570 INFO:teuthology.orchestra.run.trial042.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-22T19:47:06.570 INFO:teuthology.orchestra.run.trial042.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-22T19:47:06.570 INFO:teuthology.orchestra.run.trial042.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-22T19:47:06.570 INFO:teuthology.orchestra.run.trial042.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout:Suggested packages: 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout:Recommended packages: 2026-02-22T19:47:06.571 INFO:teuthology.orchestra.run.trial042.stdout: btrfs-tools 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout:The following NEW packages will be installed: 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T19:47:06.605 INFO:teuthology.orchestra.run.trial042.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T19:47:06.606 INFO:teuthology.orchestra.run.trial042.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-22T19:47:06.606 INFO:teuthology.orchestra.run.trial042.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-22T19:47:06.606 INFO:teuthology.orchestra.run.trial042.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-22T19:47:06.606 INFO:teuthology.orchestra.run.trial042.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-22T19:47:06.606 INFO:teuthology.orchestra.run.trial042.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-22T19:47:06.606 INFO:teuthology.orchestra.run.trial042.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-22T19:47:06.606 INFO:teuthology.orchestra.run.trial042.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse rbd-mirror socat 2026-02-22T19:47:06.606 INFO:teuthology.orchestra.run.trial042.stdout: unzip xmlstarlet zip 2026-02-22T19:47:06.643 INFO:teuthology.orchestra.run.trial066.stdout:Building dependency tree... 2026-02-22T19:47:06.644 INFO:teuthology.orchestra.run.trial066.stdout:Reading state information... 2026-02-22T19:47:06.737 INFO:teuthology.orchestra.run.trial066.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T19:47:06.737 INFO:teuthology.orchestra.run.trial066.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T19:47:06.737 INFO:teuthology.orchestra.run.trial066.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T19:47:06.737 INFO:teuthology.orchestra.run.trial066.stdout: sg3-utils-udev 2026-02-22T19:47:06.737 INFO:teuthology.orchestra.run.trial066.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T19:47:06.737 INFO:teuthology.orchestra.run.trial066.stdout:The following additional packages will be installed: 2026-02-22T19:47:06.737 INFO:teuthology.orchestra.run.trial066.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-22T19:47:06.737 INFO:teuthology.orchestra.run.trial066.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-22T19:47:06.738 INFO:teuthology.orchestra.run.trial066.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout:Suggested packages: 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout:Recommended packages: 2026-02-22T19:47:06.739 INFO:teuthology.orchestra.run.trial066.stdout: btrfs-tools 2026-02-22T19:47:06.772 INFO:teuthology.orchestra.run.trial066.stdout:The following NEW packages will be installed: 2026-02-22T19:47:06.772 INFO:teuthology.orchestra.run.trial066.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-22T19:47:06.772 INFO:teuthology.orchestra.run.trial066.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-22T19:47:06.772 INFO:teuthology.orchestra.run.trial066.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse rbd-mirror socat 2026-02-22T19:47:06.773 INFO:teuthology.orchestra.run.trial066.stdout: unzip xmlstarlet zip 2026-02-22T19:47:06.956 INFO:teuthology.orchestra.run.trial042.stdout:0 upgraded, 109 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T19:47:06.956 INFO:teuthology.orchestra.run.trial042.stdout:Need to get 181 MB of archives. 2026-02-22T19:47:06.956 INFO:teuthology.orchestra.run.trial042.stdout:After this operation, 821 MB of additional disk space will be used. 2026-02-22T19:47:06.956 INFO:teuthology.orchestra.run.trial042.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-22T19:47:07.037 INFO:teuthology.orchestra.run.trial042.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-689-gf90f4eae-1jammy [3597 kB] 2026-02-22T19:47:07.128 INFO:teuthology.orchestra.run.trial066.stdout:0 upgraded, 109 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T19:47:07.128 INFO:teuthology.orchestra.run.trial066.stdout:Need to get 181 MB of archives. 2026-02-22T19:47:07.128 INFO:teuthology.orchestra.run.trial066.stdout:After this operation, 821 MB of additional disk space will be used. 2026-02-22T19:47:07.128 INFO:teuthology.orchestra.run.trial066.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-689-gf90f4eae-1jammy [3597 kB] 2026-02-22T19:47:07.190 INFO:teuthology.orchestra.run.trial066.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-22T19:47:07.292 INFO:teuthology.orchestra.run.trial042.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-22T19:47:07.303 INFO:teuthology.orchestra.run.trial042.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-22T19:47:07.373 INFO:teuthology.orchestra.run.trial042.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-22T19:47:07.574 INFO:teuthology.orchestra.run.trial042.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-22T19:47:07.583 INFO:teuthology.orchestra.run.trial042.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-22T19:47:07.599 INFO:teuthology.orchestra.run.trial066.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-22T19:47:07.610 INFO:teuthology.orchestra.run.trial042.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-22T19:47:07.612 INFO:teuthology.orchestra.run.trial066.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-22T19:47:07.618 INFO:teuthology.orchestra.run.trial042.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-22T19:47:07.621 INFO:teuthology.orchestra.run.trial042.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-22T19:47:07.621 INFO:teuthology.orchestra.run.trial042.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-22T19:47:07.622 INFO:teuthology.orchestra.run.trial042.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-22T19:47:07.639 INFO:teuthology.orchestra.run.trial042.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-22T19:47:07.643 INFO:teuthology.orchestra.run.trial042.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-22T19:47:07.646 INFO:teuthology.orchestra.run.trial042.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-22T19:47:07.697 INFO:teuthology.orchestra.run.trial066.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-22T19:47:07.714 INFO:teuthology.orchestra.run.trial042.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-22T19:47:07.714 INFO:teuthology.orchestra.run.trial042.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-22T19:47:07.717 INFO:teuthology.orchestra.run.trial042.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-22T19:47:07.719 INFO:teuthology.orchestra.run.trial042.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-22T19:47:07.720 INFO:teuthology.orchestra.run.trial042.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-22T19:47:07.721 INFO:teuthology.orchestra.run.trial042.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-22T19:47:07.721 INFO:teuthology.orchestra.run.trial042.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-22T19:47:07.722 INFO:teuthology.orchestra.run.trial042.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-22T19:47:07.792 INFO:teuthology.orchestra.run.trial042.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-22T19:47:07.793 INFO:teuthology.orchestra.run.trial042.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-22T19:47:07.793 INFO:teuthology.orchestra.run.trial042.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-22T19:47:07.793 INFO:teuthology.orchestra.run.trial042.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-22T19:47:07.793 INFO:teuthology.orchestra.run.trial042.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-22T19:47:07.864 INFO:teuthology.orchestra.run.trial042.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-22T19:47:07.866 INFO:teuthology.orchestra.run.trial042.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-22T19:47:07.867 INFO:teuthology.orchestra.run.trial042.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-22T19:47:07.867 INFO:teuthology.orchestra.run.trial042.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-22T19:47:07.868 INFO:teuthology.orchestra.run.trial042.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-22T19:47:07.937 INFO:teuthology.orchestra.run.trial042.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-22T19:47:07.937 INFO:teuthology.orchestra.run.trial042.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-22T19:47:07.938 INFO:teuthology.orchestra.run.trial042.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-22T19:47:07.938 INFO:teuthology.orchestra.run.trial042.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-22T19:47:07.938 INFO:teuthology.orchestra.run.trial042.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-22T19:47:07.942 INFO:teuthology.orchestra.run.trial066.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-22T19:47:07.955 INFO:teuthology.orchestra.run.trial066.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-22T19:47:07.987 INFO:teuthology.orchestra.run.trial066.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-22T19:47:07.997 INFO:teuthology.orchestra.run.trial066.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-22T19:47:07.999 INFO:teuthology.orchestra.run.trial066.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-22T19:47:08.000 INFO:teuthology.orchestra.run.trial066.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-22T19:47:08.001 INFO:teuthology.orchestra.run.trial066.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-22T19:47:08.008 INFO:teuthology.orchestra.run.trial042.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-22T19:47:08.008 INFO:teuthology.orchestra.run.trial042.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-22T19:47:08.009 INFO:teuthology.orchestra.run.trial042.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-22T19:47:08.009 INFO:teuthology.orchestra.run.trial042.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-22T19:47:08.010 INFO:teuthology.orchestra.run.trial042.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-22T19:47:08.021 INFO:teuthology.orchestra.run.trial066.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-22T19:47:08.026 INFO:teuthology.orchestra.run.trial066.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-22T19:47:08.031 INFO:teuthology.orchestra.run.trial066.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-22T19:47:08.081 INFO:teuthology.orchestra.run.trial042.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-22T19:47:08.082 INFO:teuthology.orchestra.run.trial042.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-22T19:47:08.083 INFO:teuthology.orchestra.run.trial042.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-22T19:47:08.083 INFO:teuthology.orchestra.run.trial042.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-22T19:47:08.084 INFO:teuthology.orchestra.run.trial042.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-22T19:47:08.114 INFO:teuthology.orchestra.run.trial066.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-22T19:47:08.114 INFO:teuthology.orchestra.run.trial066.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-22T19:47:08.116 INFO:teuthology.orchestra.run.trial066.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-22T19:47:08.119 INFO:teuthology.orchestra.run.trial066.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-22T19:47:08.121 INFO:teuthology.orchestra.run.trial066.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-22T19:47:08.121 INFO:teuthology.orchestra.run.trial066.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-22T19:47:08.122 INFO:teuthology.orchestra.run.trial066.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-22T19:47:08.122 INFO:teuthology.orchestra.run.trial066.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-22T19:47:08.152 INFO:teuthology.orchestra.run.trial042.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-22T19:47:08.155 INFO:teuthology.orchestra.run.trial042.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-22T19:47:08.155 INFO:teuthology.orchestra.run.trial042.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-22T19:47:08.175 INFO:teuthology.orchestra.run.trial042.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-22T19:47:08.175 INFO:teuthology.orchestra.run.trial042.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-22T19:47:08.208 INFO:teuthology.orchestra.run.trial066.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-22T19:47:08.208 INFO:teuthology.orchestra.run.trial066.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-22T19:47:08.209 INFO:teuthology.orchestra.run.trial066.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-22T19:47:08.209 INFO:teuthology.orchestra.run.trial066.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-22T19:47:08.209 INFO:teuthology.orchestra.run.trial066.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-22T19:47:08.224 INFO:teuthology.orchestra.run.trial042.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-22T19:47:08.224 INFO:teuthology.orchestra.run.trial042.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-22T19:47:08.225 INFO:teuthology.orchestra.run.trial042.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-22T19:47:08.225 INFO:teuthology.orchestra.run.trial042.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-22T19:47:08.229 INFO:teuthology.orchestra.run.trial042.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-22T19:47:08.296 INFO:teuthology.orchestra.run.trial042.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-22T19:47:08.297 INFO:teuthology.orchestra.run.trial066.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-22T19:47:08.297 INFO:teuthology.orchestra.run.trial042.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-22T19:47:08.298 INFO:teuthology.orchestra.run.trial042.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-22T19:47:08.299 INFO:teuthology.orchestra.run.trial066.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-22T19:47:08.299 INFO:teuthology.orchestra.run.trial066.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-22T19:47:08.299 INFO:teuthology.orchestra.run.trial066.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-22T19:47:08.300 INFO:teuthology.orchestra.run.trial066.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-22T19:47:08.301 INFO:teuthology.orchestra.run.trial042.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-22T19:47:08.304 INFO:teuthology.orchestra.run.trial042.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-22T19:47:08.355 INFO:teuthology.orchestra.run.trial066.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-689-gf90f4eae-1jammy [3256 kB] 2026-02-22T19:47:08.368 INFO:teuthology.orchestra.run.trial042.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-22T19:47:08.368 INFO:teuthology.orchestra.run.trial042.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-22T19:47:08.373 INFO:teuthology.orchestra.run.trial042.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-22T19:47:08.373 INFO:teuthology.orchestra.run.trial042.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-22T19:47:08.377 INFO:teuthology.orchestra.run.trial042.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-22T19:47:08.384 INFO:teuthology.orchestra.run.trial066.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-22T19:47:08.384 INFO:teuthology.orchestra.run.trial066.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-22T19:47:08.385 INFO:teuthology.orchestra.run.trial066.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-22T19:47:08.385 INFO:teuthology.orchestra.run.trial066.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-22T19:47:08.386 INFO:teuthology.orchestra.run.trial066.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-22T19:47:08.440 INFO:teuthology.orchestra.run.trial042.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-22T19:47:08.440 INFO:teuthology.orchestra.run.trial042.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-22T19:47:08.441 INFO:teuthology.orchestra.run.trial042.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-22T19:47:08.442 INFO:teuthology.orchestra.run.trial042.stdout:Get:72 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-22T19:47:08.443 INFO:teuthology.orchestra.run.trial042.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-22T19:47:08.472 INFO:teuthology.orchestra.run.trial066.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-22T19:47:08.472 INFO:teuthology.orchestra.run.trial066.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-22T19:47:08.472 INFO:teuthology.orchestra.run.trial066.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-22T19:47:08.472 INFO:teuthology.orchestra.run.trial066.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-22T19:47:08.473 INFO:teuthology.orchestra.run.trial066.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-22T19:47:08.513 INFO:teuthology.orchestra.run.trial042.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-22T19:47:08.513 INFO:teuthology.orchestra.run.trial042.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-22T19:47:08.513 INFO:teuthology.orchestra.run.trial042.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-22T19:47:08.515 INFO:teuthology.orchestra.run.trial042.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-22T19:47:08.515 INFO:teuthology.orchestra.run.trial042.stdout:Get:78 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-22T19:47:08.559 INFO:teuthology.orchestra.run.trial066.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-22T19:47:08.560 INFO:teuthology.orchestra.run.trial066.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-22T19:47:08.563 INFO:teuthology.orchestra.run.trial066.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-22T19:47:08.563 INFO:teuthology.orchestra.run.trial066.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-22T19:47:08.564 INFO:teuthology.orchestra.run.trial066.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-22T19:47:08.647 INFO:teuthology.orchestra.run.trial066.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-22T19:47:08.650 INFO:teuthology.orchestra.run.trial066.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-22T19:47:08.650 INFO:teuthology.orchestra.run.trial066.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-22T19:47:08.675 INFO:teuthology.orchestra.run.trial066.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-22T19:47:08.675 INFO:teuthology.orchestra.run.trial066.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-22T19:47:08.735 INFO:teuthology.orchestra.run.trial066.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-22T19:47:08.735 INFO:teuthology.orchestra.run.trial066.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-22T19:47:08.735 INFO:teuthology.orchestra.run.trial066.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-22T19:47:08.736 INFO:teuthology.orchestra.run.trial066.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-22T19:47:08.740 INFO:teuthology.orchestra.run.trial066.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-22T19:47:08.823 INFO:teuthology.orchestra.run.trial066.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-22T19:47:08.824 INFO:teuthology.orchestra.run.trial066.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-22T19:47:08.825 INFO:teuthology.orchestra.run.trial066.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-22T19:47:08.829 INFO:teuthology.orchestra.run.trial066.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-22T19:47:08.832 INFO:teuthology.orchestra.run.trial066.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-22T19:47:08.911 INFO:teuthology.orchestra.run.trial066.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-22T19:47:08.911 INFO:teuthology.orchestra.run.trial066.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-22T19:47:08.917 INFO:teuthology.orchestra.run.trial066.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-22T19:47:08.917 INFO:teuthology.orchestra.run.trial066.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-22T19:47:08.921 INFO:teuthology.orchestra.run.trial066.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-22T19:47:08.998 INFO:teuthology.orchestra.run.trial066.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-22T19:47:08.999 INFO:teuthology.orchestra.run.trial066.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-22T19:47:08.999 INFO:teuthology.orchestra.run.trial066.stdout:Get:72 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-22T19:47:09.001 INFO:teuthology.orchestra.run.trial066.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-22T19:47:09.002 INFO:teuthology.orchestra.run.trial066.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-22T19:47:09.086 INFO:teuthology.orchestra.run.trial066.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-22T19:47:09.086 INFO:teuthology.orchestra.run.trial066.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-22T19:47:09.087 INFO:teuthology.orchestra.run.trial066.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-22T19:47:09.089 INFO:teuthology.orchestra.run.trial066.stdout:Get:78 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-22T19:47:09.090 INFO:teuthology.orchestra.run.trial066.stdout:Get:79 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-22T19:47:09.576 INFO:teuthology.orchestra.run.trial066.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-689-gf90f4eae-1jammy [979 kB] 2026-02-22T19:47:10.042 INFO:teuthology.orchestra.run.trial066.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-689-gf90f4eae-1jammy [357 kB] 2026-02-22T19:47:10.238 INFO:teuthology.orchestra.run.trial066.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-689-gf90f4eae-1jammy [32.9 kB] 2026-02-22T19:47:10.239 INFO:teuthology.orchestra.run.trial066.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-689-gf90f4eae-1jammy [184 kB] 2026-02-22T19:47:10.371 INFO:teuthology.orchestra.run.trial066.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-689-gf90f4eae-1jammy [70.0 kB] 2026-02-22T19:47:10.374 INFO:teuthology.orchestra.run.trial066.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-689-gf90f4eae-1jammy [335 kB] 2026-02-22T19:47:10.580 INFO:teuthology.orchestra.run.trial066.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-689-gf90f4eae-1jammy [6935 kB] 2026-02-22T19:47:16.401 INFO:teuthology.orchestra.run.trial042.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-689-gf90f4eae-1jammy [3256 kB] 2026-02-22T19:47:17.616 INFO:teuthology.orchestra.run.trial066.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-689-gf90f4eae-1jammy [112 kB] 2026-02-22T19:47:17.867 INFO:teuthology.orchestra.run.trial066.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-689-gf90f4eae-1jammy [470 kB] 2026-02-22T19:47:18.915 INFO:teuthology.orchestra.run.trial066.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-689-gf90f4eae-1jammy [26.5 MB] 2026-02-22T19:47:22.478 INFO:teuthology.orchestra.run.trial042.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-689-gf90f4eae-1jammy [979 kB] 2026-02-22T19:47:24.268 INFO:teuthology.orchestra.run.trial042.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-689-gf90f4eae-1jammy [357 kB] 2026-02-22T19:47:24.855 INFO:teuthology.orchestra.run.trial042.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-689-gf90f4eae-1jammy [32.9 kB] 2026-02-22T19:47:24.855 INFO:teuthology.orchestra.run.trial042.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-689-gf90f4eae-1jammy [184 kB] 2026-02-22T19:47:25.191 INFO:teuthology.orchestra.run.trial042.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-689-gf90f4eae-1jammy [70.0 kB] 2026-02-22T19:47:25.297 INFO:teuthology.orchestra.run.trial042.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-689-gf90f4eae-1jammy [335 kB] 2026-02-22T19:47:25.810 INFO:teuthology.orchestra.run.trial042.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-689-gf90f4eae-1jammy [6935 kB] 2026-02-22T19:47:39.045 INFO:teuthology.orchestra.run.trial042.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-689-gf90f4eae-1jammy [112 kB] 2026-02-22T19:47:39.247 INFO:teuthology.orchestra.run.trial042.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-689-gf90f4eae-1jammy [470 kB] 2026-02-22T19:47:40.070 INFO:teuthology.orchestra.run.trial042.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-689-gf90f4eae-1jammy [26.5 MB] 2026-02-22T19:48:06.167 INFO:teuthology.orchestra.run.trial066.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-689-gf90f4eae-1jammy [5178 kB] 2026-02-22T19:48:18.046 INFO:teuthology.orchestra.run.trial066.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-689-gf90f4eae-1jammy [248 kB] 2026-02-22T19:48:18.822 INFO:teuthology.orchestra.run.trial066.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-689-gf90f4eae-1jammy [125 kB] 2026-02-22T19:48:19.150 INFO:teuthology.orchestra.run.trial066.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-689-gf90f4eae-1jammy [1081 kB] 2026-02-22T19:48:22.706 INFO:teuthology.orchestra.run.trial066.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-689-gf90f4eae-1jammy [6239 kB] 2026-02-22T19:48:35.135 INFO:teuthology.orchestra.run.trial042.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-689-gf90f4eae-1jammy [5178 kB] 2026-02-22T19:48:42.616 INFO:teuthology.orchestra.run.trial066.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-689-gf90f4eae-1jammy [23.0 MB] 2026-02-22T19:48:48.504 INFO:teuthology.orchestra.run.trial042.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-689-gf90f4eae-1jammy [248 kB] 2026-02-22T19:48:49.262 INFO:teuthology.orchestra.run.trial042.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-689-gf90f4eae-1jammy [125 kB] 2026-02-22T19:48:49.651 INFO:teuthology.orchestra.run.trial042.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-689-gf90f4eae-1jammy [1081 kB] 2026-02-22T19:48:52.057 INFO:teuthology.orchestra.run.trial042.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-689-gf90f4eae-1jammy [6239 kB] 2026-02-22T19:49:04.133 INFO:teuthology.orchestra.run.trial042.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-689-gf90f4eae-1jammy [23.0 MB] 2026-02-22T19:49:29.355 INFO:teuthology.orchestra.run.trial066.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-689-gf90f4eae-1jammy [14.2 kB] 2026-02-22T19:49:29.386 INFO:teuthology.orchestra.run.trial066.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-689-gf90f4eae-1jammy [1173 kB] 2026-02-22T19:49:30.951 INFO:teuthology.orchestra.run.trial066.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-689-gf90f4eae-1jammy [2503 kB] 2026-02-22T19:49:35.231 INFO:teuthology.orchestra.run.trial066.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-689-gf90f4eae-1jammy [798 kB] 2026-02-22T19:49:36.345 INFO:teuthology.orchestra.run.trial066.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-689-gf90f4eae-1jammy [157 kB] 2026-02-22T19:49:36.522 INFO:teuthology.orchestra.run.trial066.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-689-gf90f4eae-1jammy [2396 kB] 2026-02-22T19:49:39.924 INFO:teuthology.orchestra.run.trial066.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-689-gf90f4eae-1jammy [8625 kB] 2026-02-22T19:49:41.101 INFO:teuthology.orchestra.run.trial042.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-689-gf90f4eae-1jammy [14.2 kB] 2026-02-22T19:49:41.168 INFO:teuthology.orchestra.run.trial042.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-689-gf90f4eae-1jammy [1173 kB] 2026-02-22T19:49:43.634 INFO:teuthology.orchestra.run.trial042.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-689-gf90f4eae-1jammy [2503 kB] 2026-02-22T19:49:47.025 INFO:teuthology.orchestra.run.trial042.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-689-gf90f4eae-1jammy [798 kB] 2026-02-22T19:49:48.086 INFO:teuthology.orchestra.run.trial042.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-689-gf90f4eae-1jammy [157 kB] 2026-02-22T19:49:48.414 INFO:teuthology.orchestra.run.trial042.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-689-gf90f4eae-1jammy [2396 kB] 2026-02-22T19:49:52.271 INFO:teuthology.orchestra.run.trial042.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-689-gf90f4eae-1jammy [8625 kB] 2026-02-22T19:49:53.635 INFO:teuthology.orchestra.run.trial066.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-689-gf90f4eae-1jammy [14.3 kB] 2026-02-22T19:49:53.635 INFO:teuthology.orchestra.run.trial066.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-689-gf90f4eae-1jammy [52.1 MB] 2026-02-22T19:50:03.958 INFO:teuthology.orchestra.run.trial042.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-689-gf90f4eae-1jammy [14.3 kB] 2026-02-22T19:50:03.959 INFO:teuthology.orchestra.run.trial042.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-689-gf90f4eae-1jammy [52.1 MB] 2026-02-22T19:50:47.394 INFO:teuthology.orchestra.run.trial066.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-689-gf90f4eae-1jammy [135 kB] 2026-02-22T19:50:47.445 INFO:teuthology.orchestra.run.trial066.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-689-gf90f4eae-1jammy [41.0 kB] 2026-02-22T19:50:47.448 INFO:teuthology.orchestra.run.trial066.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-689-gf90f4eae-1jammy [13.7 MB] 2026-02-22T19:50:47.689 INFO:teuthology.orchestra.run.trial042.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-689-gf90f4eae-1jammy [135 kB] 2026-02-22T19:50:47.718 INFO:teuthology.orchestra.run.trial042.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-689-gf90f4eae-1jammy [41.0 kB] 2026-02-22T19:50:47.722 INFO:teuthology.orchestra.run.trial042.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-689-gf90f4eae-1jammy [13.7 MB] 2026-02-22T19:50:51.501 INFO:teuthology.orchestra.run.trial042.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-689-gf90f4eae-1jammy [92.3 kB] 2026-02-22T19:50:51.532 INFO:teuthology.orchestra.run.trial042.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.2.3-689-gf90f4eae-1jammy [3151 kB] 2026-02-22T19:50:52.628 INFO:teuthology.orchestra.run.trial042.stdout:Fetched 181 MB in 3min 46s (801 kB/s) 2026-02-22T19:50:52.644 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-22T19:50:52.800 INFO:teuthology.orchestra.run.trial042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-22T19:50:52.801 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-22T19:50:52.803 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T19:50:52.824 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-22T19:50:52.837 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-22T19:50:52.837 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T19:50:52.854 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-22T19:50:52.867 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-22T19:50:52.867 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T19:50:52.885 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-22T19:50:52.893 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:52.898 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:52.934 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-22T19:50:52.941 INFO:teuthology.orchestra.run.trial066.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-689-gf90f4eae-1jammy [92.3 kB] 2026-02-22T19:50:52.942 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:52.943 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:52.959 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-22T19:50:52.967 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:52.967 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:52.980 INFO:teuthology.orchestra.run.trial066.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.2.3-689-gf90f4eae-1jammy [3151 kB] 2026-02-22T19:50:52.987 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-22T19:50:52.995 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-22T19:50:52.995 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T19:50:53.012 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package librados2. 2026-02-22T19:50:53.020 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../007-librados2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.020 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.060 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libnbd0. 2026-02-22T19:50:53.068 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-22T19:50:53.069 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-22T19:50:53.083 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package librbd1. 2026-02-22T19:50:53.091 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../009-librbd1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.091 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.132 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libcephfs2. 2026-02-22T19:50:53.141 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.141 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.162 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-rados. 2026-02-22T19:50:53.170 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../011-python3-rados_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.170 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.186 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-22T19:50:53.194 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:53.195 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.208 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-cephfs. 2026-02-22T19:50:53.216 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.216 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.232 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-22T19:50:53.240 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:53.241 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.257 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-22T19:50:53.265 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-22T19:50:53.265 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T19:50:53.281 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-prettytable. 2026-02-22T19:50:53.289 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-22T19:50:53.289 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-22T19:50:53.303 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-rbd. 2026-02-22T19:50:53.311 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.311 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.329 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-22T19:50:53.337 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-22T19:50:53.337 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T19:50:53.355 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-22T19:50:53.363 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-22T19:50:53.364 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T19:50:53.381 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-22T19:50:53.389 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-22T19:50:53.389 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T19:50:53.406 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package lua5.1. 2026-02-22T19:50:53.414 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-22T19:50:53.414 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-22T19:50:53.434 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package lua-any. 2026-02-22T19:50:53.442 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-22T19:50:53.442 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-22T19:50:53.456 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package zip. 2026-02-22T19:50:53.464 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-22T19:50:53.464 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking zip (3.0-12build2) ... 2026-02-22T19:50:53.482 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package unzip. 2026-02-22T19:50:53.490 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-22T19:50:53.490 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-22T19:50:53.508 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package luarocks. 2026-02-22T19:50:53.516 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-22T19:50:53.517 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-22T19:50:53.554 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package librgw2. 2026-02-22T19:50:53.562 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../026-librgw2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.562 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.636 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-rgw. 2026-02-22T19:50:53.645 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.645 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.661 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-22T19:50:53.669 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-22T19:50:53.669 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T19:50:53.683 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libradosstriper1. 2026-02-22T19:50:53.691 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.692 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.710 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-common. 2026-02-22T19:50:53.718 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../030-ceph-common_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.718 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:53.925 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-base. 2026-02-22T19:50:53.933 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../031-ceph-base_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:53.938 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.002 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-22T19:50:54.011 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-22T19:50:54.011 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-22T19:50:54.025 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-cheroot. 2026-02-22T19:50:54.033 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-22T19:50:54.034 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T19:50:54.050 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-22T19:50:54.058 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-22T19:50:54.058 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-22T19:50:54.072 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-22T19:50:54.081 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-22T19:50:54.081 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-22T19:50:54.094 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-22T19:50:54.103 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-22T19:50:54.103 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-22T19:50:54.117 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-tempora. 2026-02-22T19:50:54.125 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-22T19:50:54.125 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-22T19:50:54.139 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-portend. 2026-02-22T19:50:54.147 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-22T19:50:54.147 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-22T19:50:54.162 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-22T19:50:54.170 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-22T19:50:54.171 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-22T19:50:54.185 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-22T19:50:54.193 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-22T19:50:54.193 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-22T19:50:54.205 INFO:teuthology.orchestra.run.trial066.stdout:Fetched 181 MB in 3min 47s (796 kB/s) 2026-02-22T19:50:54.216 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-natsort. 2026-02-22T19:50:54.222 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-22T19:50:54.224 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-22T19:50:54.224 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-22T19:50:54.240 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-logutils. 2026-02-22T19:50:54.249 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-22T19:50:54.249 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-22T19:50:54.264 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-mako. 2026-02-22T19:50:54.272 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-22T19:50:54.272 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T19:50:54.289 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-22T19:50:54.298 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-22T19:50:54.298 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-22T19:50:54.312 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-22T19:50:54.320 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-22T19:50:54.320 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-22T19:50:54.334 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-webob. 2026-02-22T19:50:54.342 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-22T19:50:54.342 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T19:50:54.359 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-waitress. 2026-02-22T19:50:54.367 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-22T19:50:54.369 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T19:50:54.377 INFO:teuthology.orchestra.run.trial066.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-22T19:50:54.379 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-22T19:50:54.381 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T19:50:54.385 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-tempita. 2026-02-22T19:50:54.393 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-22T19:50:54.393 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T19:50:54.398 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-22T19:50:54.407 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-22T19:50:54.407 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T19:50:54.408 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-paste. 2026-02-22T19:50:54.416 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-22T19:50:54.416 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T19:50:54.423 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-22T19:50:54.435 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-22T19:50:54.435 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T19:50:54.444 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-22T19:50:54.452 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-22T19:50:54.452 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T19:50:54.453 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-22T19:50:54.461 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:54.465 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:54.466 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-22T19:50:54.474 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-22T19:50:54.475 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-22T19:50:54.490 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-webtest. 2026-02-22T19:50:54.494 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-22T19:50:54.498 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-22T19:50:54.499 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-22T19:50:54.502 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:54.502 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:54.514 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-pecan. 2026-02-22T19:50:54.519 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-22T19:50:54.522 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-22T19:50:54.522 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T19:50:54.527 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:54.527 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:54.547 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-22T19:50:54.547 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-22T19:50:54.555 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-22T19:50:54.555 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T19:50:54.555 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-22T19:50:54.556 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T19:50:54.571 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package librados2. 2026-02-22T19:50:54.576 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-22T19:50:54.579 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../007-librados2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.580 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.584 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:54.584 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.613 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-22T19:50:54.619 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libnbd0. 2026-02-22T19:50:54.621 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.621 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.628 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-22T19:50:54.628 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-22T19:50:54.636 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-mgr. 2026-02-22T19:50:54.643 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package librbd1. 2026-02-22T19:50:54.645 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.645 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.651 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../009-librbd1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.652 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.667 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-mon. 2026-02-22T19:50:54.676 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.676 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.693 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libcephfs2. 2026-02-22T19:50:54.701 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.701 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.721 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-rados. 2026-02-22T19:50:54.729 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../011-python3-rados_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.730 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.746 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-22T19:50:54.748 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-22T19:50:54.755 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:54.755 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.757 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-22T19:50:54.757 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T19:50:54.768 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-cephfs. 2026-02-22T19:50:54.773 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-osd. 2026-02-22T19:50:54.776 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.776 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.781 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.781 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.791 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-22T19:50:54.800 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:54.800 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.816 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-22T19:50:54.824 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-22T19:50:54.824 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T19:50:54.840 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-prettytable. 2026-02-22T19:50:54.848 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-22T19:50:54.849 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-22T19:50:54.863 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-rbd. 2026-02-22T19:50:54.871 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.871 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.888 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-22T19:50:54.896 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-22T19:50:54.897 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T19:50:54.914 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-22T19:50:54.922 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-22T19:50:54.923 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T19:50:54.936 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph. 2026-02-22T19:50:54.938 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-22T19:50:54.944 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../061-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.945 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.946 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-22T19:50:54.947 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T19:50:54.959 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-fuse. 2026-02-22T19:50:54.964 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package lua5.1. 2026-02-22T19:50:54.967 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:54.967 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:54.971 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-22T19:50:54.972 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-22T19:50:54.991 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package lua-any. 2026-02-22T19:50:54.991 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-mds. 2026-02-22T19:50:54.999 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-22T19:50:54.999 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-22T19:50:55.000 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.000 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.014 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package zip. 2026-02-22T19:50:55.022 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-22T19:50:55.022 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking zip (3.0-12build2) ... 2026-02-22T19:50:55.034 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package cephadm. 2026-02-22T19:50:55.038 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package unzip. 2026-02-22T19:50:55.042 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../064-cephadm_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.042 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.046 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-22T19:50:55.046 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-22T19:50:55.058 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-22T19:50:55.063 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package luarocks. 2026-02-22T19:50:55.066 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T19:50:55.067 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:55.071 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-22T19:50:55.071 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-22T19:50:55.087 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-22T19:50:55.096 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:55.096 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.108 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package librgw2. 2026-02-22T19:50:55.116 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-22T19:50:55.116 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../026-librgw2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.117 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.124 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-22T19:50:55.124 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-22T19:50:55.139 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-routes. 2026-02-22T19:50:55.147 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-22T19:50:55.147 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T19:50:55.167 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-22T19:50:55.175 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:55.175 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.190 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-rgw. 2026-02-22T19:50:55.198 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.199 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.213 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-22T19:50:55.221 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-22T19:50:55.221 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T19:50:55.235 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libradosstriper1. 2026-02-22T19:50:55.243 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.243 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.262 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-common. 2026-02-22T19:50:55.269 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../030-ceph-common_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.270 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.381 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-22T19:50:55.389 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-22T19:50:55.389 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T19:50:55.428 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-joblib. 2026-02-22T19:50:55.435 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-22T19:50:55.435 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T19:50:55.462 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-22T19:50:55.470 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-22T19:50:55.470 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-22T19:50:55.476 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-base. 2026-02-22T19:50:55.485 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../031-ceph-base_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.485 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-sklearn. 2026-02-22T19:50:55.490 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.493 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-22T19:50:55.494 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T19:50:55.552 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-22T19:50:55.560 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-22T19:50:55.560 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-22T19:50:55.574 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-22T19:50:55.574 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-cheroot. 2026-02-22T19:50:55.582 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-22T19:50:55.583 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T19:50:55.583 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:55.583 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.599 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-22T19:50:55.607 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-22T19:50:55.608 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-22T19:50:55.622 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-22T19:50:55.631 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-22T19:50:55.631 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-22T19:50:55.645 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-22T19:50:55.653 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-22T19:50:55.653 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-22T19:50:55.668 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-tempora. 2026-02-22T19:50:55.676 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-22T19:50:55.676 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-22T19:50:55.691 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-portend. 2026-02-22T19:50:55.700 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-22T19:50:55.700 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-22T19:50:55.714 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-22T19:50:55.723 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-22T19:50:55.723 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-22T19:50:55.737 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-22T19:50:55.745 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-22T19:50:55.746 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-22T19:50:55.752 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-cachetools. 2026-02-22T19:50:55.761 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-22T19:50:55.761 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-22T19:50:55.768 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-natsort. 2026-02-22T19:50:55.776 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-rsa. 2026-02-22T19:50:55.777 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-22T19:50:55.777 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-22T19:50:55.784 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-22T19:50:55.785 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-22T19:50:55.793 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-logutils. 2026-02-22T19:50:55.801 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-22T19:50:55.801 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-22T19:50:55.802 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-google-auth. 2026-02-22T19:50:55.810 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-22T19:50:55.811 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-22T19:50:55.815 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-mako. 2026-02-22T19:50:55.823 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-22T19:50:55.824 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T19:50:55.828 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-22T19:50:55.836 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-22T19:50:55.837 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T19:50:55.841 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-22T19:50:55.849 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-22T19:50:55.849 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-22T19:50:55.852 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-websocket. 2026-02-22T19:50:55.861 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-22T19:50:55.861 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-22T19:50:55.863 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-22T19:50:55.871 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-22T19:50:55.872 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-22T19:50:55.878 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-22T19:50:55.885 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-webob. 2026-02-22T19:50:55.887 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-22T19:50:55.887 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T19:50:55.893 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-22T19:50:55.894 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T19:50:55.909 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-waitress. 2026-02-22T19:50:55.917 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-22T19:50:55.920 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T19:50:55.935 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-tempita. 2026-02-22T19:50:55.943 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-22T19:50:55.943 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T19:50:55.958 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-paste. 2026-02-22T19:50:55.966 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-22T19:50:55.967 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T19:50:55.983 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-22T19:50:55.992 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:55.992 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.994 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-22T19:50:56.002 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-22T19:50:56.002 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T19:50:56.006 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-22T19:50:56.014 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-22T19:50:56.016 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-22T19:50:56.021 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T19:50:56.024 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-22T19:50:56.024 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-22T19:50:56.037 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-22T19:50:56.040 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-webtest. 2026-02-22T19:50:56.045 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T19:50:56.046 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:56.048 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-22T19:50:56.048 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-22T19:50:56.061 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package jq. 2026-02-22T19:50:56.064 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-pecan. 2026-02-22T19:50:56.069 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T19:50:56.069 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:56.072 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-22T19:50:56.072 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T19:50:56.083 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package socat. 2026-02-22T19:50:56.092 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-22T19:50:56.092 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-22T19:50:56.097 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-22T19:50:56.104 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-22T19:50:56.105 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T19:50:56.112 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package xmlstarlet. 2026-02-22T19:50:56.120 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-22T19:50:56.120 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-22T19:50:56.124 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-22T19:50:56.132 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:56.133 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.157 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-test. 2026-02-22T19:50:56.161 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-22T19:50:56.165 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../087-ceph-test_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.165 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.169 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.169 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.184 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-mgr. 2026-02-22T19:50:56.192 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.193 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.215 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-mon. 2026-02-22T19:50:56.223 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.223 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.278 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-22T19:50:56.287 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-22T19:50:56.287 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T19:50:56.303 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-osd. 2026-02-22T19:50:56.311 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.312 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.462 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph. 2026-02-22T19:50:56.471 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../061-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.471 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.485 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-fuse. 2026-02-22T19:50:56.493 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.493 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.516 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-mds. 2026-02-22T19:50:56.525 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.525 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.558 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package cephadm. 2026-02-22T19:50:56.566 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../064-cephadm_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.567 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.582 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-22T19:50:56.591 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T19:50:56.591 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:56.612 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-22T19:50:56.620 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:56.621 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.630 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package ceph-volume. 2026-02-22T19:50:56.638 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:56.639 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.640 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-22T19:50:56.649 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-22T19:50:56.649 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-22T19:50:56.661 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-22T19:50:56.664 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-routes. 2026-02-22T19:50:56.669 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.669 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.672 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-22T19:50:56.673 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T19:50:56.685 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-22T19:50:56.692 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-22T19:50:56.694 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-22T19:50:56.694 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T19:50:56.700 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:56.701 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.715 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-22T19:50:56.723 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-22T19:50:56.723 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-22T19:50:56.741 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package nvme-cli. 2026-02-22T19:50:56.749 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-22T19:50:56.750 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T19:50:56.782 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package pkg-config. 2026-02-22T19:50:56.790 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-22T19:50:56.790 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T19:50:56.806 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-22T19:50:56.814 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T19:50:56.815 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:56.843 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-22T19:50:56.852 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-22T19:50:56.852 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-22T19:50:56.867 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-pastescript. 2026-02-22T19:50:56.875 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-22T19:50:56.876 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-22T19:50:56.893 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-pluggy. 2026-02-22T19:50:56.901 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-22T19:50:56.902 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-22T19:50:56.904 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-22T19:50:56.913 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-22T19:50:56.913 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T19:50:56.917 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-psutil. 2026-02-22T19:50:56.925 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-22T19:50:56.925 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-22T19:50:56.944 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-py. 2026-02-22T19:50:56.952 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-22T19:50:56.952 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-joblib. 2026-02-22T19:50:56.952 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-22T19:50:56.960 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-22T19:50:56.960 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T19:50:56.972 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-pygments. 2026-02-22T19:50:56.980 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-22T19:50:56.980 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T19:50:56.988 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-22T19:50:56.996 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-22T19:50:56.997 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-22T19:50:57.012 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-sklearn. 2026-02-22T19:50:57.020 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-22T19:50:57.021 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T19:50:57.021 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-22T19:50:57.029 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-22T19:50:57.030 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-22T19:50:57.045 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-toml. 2026-02-22T19:50:57.054 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-22T19:50:57.054 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-22T19:50:57.069 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-pytest. 2026-02-22T19:50:57.077 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-22T19:50:57.077 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T19:50:57.098 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package python3-simplejson. 2026-02-22T19:50:57.102 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-22T19:50:57.107 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-22T19:50:57.107 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-22T19:50:57.111 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:57.111 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.126 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-22T19:50:57.134 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-22T19:50:57.135 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-22T19:50:57.201 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package radosgw. 2026-02-22T19:50:57.210 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../106-radosgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.210 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.275 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-cachetools. 2026-02-22T19:50:57.283 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-22T19:50:57.284 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-22T19:50:57.299 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-rsa. 2026-02-22T19:50:57.307 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-22T19:50:57.308 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-22T19:50:57.324 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package rbd-fuse. 2026-02-22T19:50:57.325 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-google-auth. 2026-02-22T19:50:57.333 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.334 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.334 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-22T19:50:57.334 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-22T19:50:57.349 INFO:teuthology.orchestra.run.trial042.stdout:Selecting previously unselected package rbd-mirror. 2026-02-22T19:50:57.352 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-22T19:50:57.357 INFO:teuthology.orchestra.run.trial042.stdout:Preparing to unpack .../108-rbd-mirror_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.358 INFO:teuthology.orchestra.run.trial042.stdout:Unpacking rbd-mirror (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.360 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-22T19:50:57.361 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T19:50:57.377 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-websocket. 2026-02-22T19:50:57.385 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-22T19:50:57.385 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-22T19:50:57.403 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-22T19:50:57.411 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-22T19:50:57.412 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T19:50:57.416 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-22T19:50:57.483 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T19:50:57.484 INFO:teuthology.orchestra.run.trial042.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T19:50:57.506 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-22T19:50:57.514 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:57.515 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.529 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-22T19:50:57.537 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-22T19:50:57.544 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T19:50:57.556 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T19:50:57.560 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-22T19:50:57.569 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T19:50:57.569 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:57.584 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package jq. 2026-02-22T19:50:57.592 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T19:50:57.593 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:57.607 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package socat. 2026-02-22T19:50:57.615 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-22T19:50:57.616 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-22T19:50:57.636 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package xmlstarlet. 2026-02-22T19:50:57.644 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-22T19:50:57.644 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-22T19:50:57.681 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-test. 2026-02-22T19:50:57.690 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../087-ceph-test_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.690 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.828 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-22T19:50:58.156 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package ceph-volume. 2026-02-22T19:50:58.164 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:58.165 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:58.188 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-22T19:50:58.189 INFO:teuthology.orchestra.run.trial042.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-22T19:50:58.196 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:58.197 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:58.200 INFO:teuthology.orchestra.run.trial042.stdout:Setting up cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:58.211 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-22T19:50:58.220 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-22T19:50:58.220 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T19:50:58.242 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-22T19:50:58.245 INFO:teuthology.orchestra.run.trial042.stdout:Adding system user cephadm....done 2026-02-22T19:50:58.250 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-22T19:50:58.250 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-22T19:50:58.255 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T19:50:58.266 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package nvme-cli. 2026-02-22T19:50:58.275 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-22T19:50:58.275 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T19:50:58.309 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package pkg-config. 2026-02-22T19:50:58.317 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-22T19:50:58.317 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T19:50:58.326 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-22T19:50:58.333 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-22T19:50:58.341 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T19:50:58.342 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:58.372 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-22T19:50:58.380 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-22T19:50:58.380 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-22T19:50:58.395 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-pastescript. 2026-02-22T19:50:58.400 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:58.401 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-22T19:50:58.403 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-22T19:50:58.403 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-22T19:50:58.422 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-pluggy. 2026-02-22T19:50:58.431 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-22T19:50:58.431 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-22T19:50:58.447 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-psutil. 2026-02-22T19:50:58.455 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-22T19:50:58.456 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-22T19:50:58.472 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-22T19:50:58.473 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-py. 2026-02-22T19:50:58.482 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-22T19:50:58.482 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-22T19:50:58.502 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-pygments. 2026-02-22T19:50:58.510 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-22T19:50:58.510 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T19:50:58.547 INFO:teuthology.orchestra.run.trial042.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T19:50:58.548 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-22T19:50:58.551 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-22T19:50:58.559 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-22T19:50:58.560 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-22T19:50:58.575 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-toml. 2026-02-22T19:50:58.583 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-22T19:50:58.584 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-22T19:50:58.599 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-pytest. 2026-02-22T19:50:58.608 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-22T19:50:58.608 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T19:50:58.630 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package python3-simplejson. 2026-02-22T19:50:58.638 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-22T19:50:58.638 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-22T19:50:58.645 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T19:50:58.656 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-22T19:50:58.664 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-22T19:50:58.665 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-22T19:50:58.733 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package radosgw. 2026-02-22T19:50:58.741 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../106-radosgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:58.742 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:58.771 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-22T19:50:58.847 INFO:teuthology.orchestra.run.trial042.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-22T19:50:58.858 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-22T19:50:58.858 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package rbd-fuse. 2026-02-22T19:50:58.867 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:58.868 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:58.883 INFO:teuthology.orchestra.run.trial066.stdout:Selecting previously unselected package rbd-mirror. 2026-02-22T19:50:58.891 INFO:teuthology.orchestra.run.trial066.stdout:Preparing to unpack .../108-rbd-mirror_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:58.892 INFO:teuthology.orchestra.run.trial066.stdout:Unpacking rbd-mirror (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:58.933 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-22T19:50:58.951 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-22T19:50:59.006 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:59.016 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T19:50:59.016 INFO:teuthology.orchestra.run.trial066.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T19:50:59.075 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T19:50:59.076 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-22T19:50:59.077 INFO:teuthology.orchestra.run.trial042.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T19:50:59.078 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T19:50:59.078 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T19:50:59.079 INFO:teuthology.orchestra.run.trial042.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-22T19:50:59.087 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T19:50:59.096 INFO:teuthology.orchestra.run.trial042.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-22T19:50:59.098 INFO:teuthology.orchestra.run.trial042.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-22T19:50:59.099 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T19:50:59.100 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-22T19:50:59.224 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-22T19:50:59.304 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T19:50:59.371 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-22T19:50:59.385 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-22T19:50:59.477 INFO:teuthology.orchestra.run.trial042.stdout:Setting up zip (3.0-12build2) ... 2026-02-22T19:50:59.478 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T19:50:59.733 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T19:50:59.771 INFO:teuthology.orchestra.run.trial066.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-22T19:50:59.781 INFO:teuthology.orchestra.run.trial066.stdout:Setting up cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:59.809 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T19:50:59.810 INFO:teuthology.orchestra.run.trial042.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-22T19:50:59.810 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T19:50:59.824 INFO:teuthology.orchestra.run.trial066.stdout:Adding system user cephadm....done 2026-02-22T19:50:59.834 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T19:50:59.905 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:59.906 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-22T19:50:59.980 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:59.981 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-22T19:51:00.030 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T19:51:00.051 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-22T19:51:00.127 INFO:teuthology.orchestra.run.trial066.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T19:51:00.128 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-22T19:51:00.157 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T19:51:00.222 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T19:51:00.241 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T19:51:00.346 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-22T19:51:00.347 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-22T19:51:00.421 INFO:teuthology.orchestra.run.trial042.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-22T19:51:00.422 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:00.422 INFO:teuthology.orchestra.run.trial066.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-22T19:51:00.434 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-22T19:51:00.506 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T19:51:00.508 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-22T19:51:00.581 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:00.650 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T19:51:00.650 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-22T19:51:00.651 INFO:teuthology.orchestra.run.trial066.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T19:51:00.652 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T19:51:00.653 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T19:51:00.654 INFO:teuthology.orchestra.run.trial066.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-22T19:51:00.671 INFO:teuthology.orchestra.run.trial066.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-22T19:51:00.673 INFO:teuthology.orchestra.run.trial066.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-22T19:51:00.674 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T19:51:00.675 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-22T19:51:00.794 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-22T19:51:00.873 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T19:51:00.950 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-22T19:51:00.998 INFO:teuthology.orchestra.run.trial042.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T19:51:01.020 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:51:01.025 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-22T19:51:01.041 INFO:teuthology.orchestra.run.trial066.stdout:Setting up zip (3.0-12build2) ... 2026-02-22T19:51:01.042 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T19:51:01.100 INFO:teuthology.orchestra.run.trial042.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T19:51:01.101 INFO:teuthology.orchestra.run.trial042.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-22T19:51:01.102 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-22T19:51:01.175 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-22T19:51:01.248 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:51:01.249 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-22T19:51:01.293 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T19:51:01.328 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-22T19:51:01.368 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T19:51:01.369 INFO:teuthology.orchestra.run.trial066.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-22T19:51:01.370 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T19:51:01.401 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-22T19:51:01.467 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T19:51:01.482 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-22T19:51:01.558 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-22T19:51:01.592 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T19:51:01.628 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-22T19:51:01.706 INFO:teuthology.orchestra.run.trial042.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T19:51:01.708 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-22T19:51:01.720 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T19:51:01.791 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T19:51:01.792 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T19:51:01.801 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T19:51:01.871 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T19:51:01.904 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-22T19:51:01.950 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T19:51:01.976 INFO:teuthology.orchestra.run.trial066.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-22T19:51:01.977 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:02.037 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-22T19:51:02.063 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T19:51:02.110 INFO:teuthology.orchestra.run.trial042.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T19:51:02.111 INFO:teuthology.orchestra.run.trial042.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-22T19:51:02.112 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T19:51:02.113 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T19:51:02.248 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-22T19:51:02.324 INFO:teuthology.orchestra.run.trial042.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-22T19:51:02.325 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-22T19:51:02.403 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:51:02.403 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-22T19:51:02.486 INFO:teuthology.orchestra.run.trial042.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-22T19:51:02.487 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-22T19:51:02.556 INFO:teuthology.orchestra.run.trial066.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T19:51:02.570 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-22T19:51:02.577 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:51:02.581 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-22T19:51:02.656 INFO:teuthology.orchestra.run.trial066.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T19:51:02.657 INFO:teuthology.orchestra.run.trial066.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-22T19:51:02.658 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-22T19:51:02.702 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-22T19:51:02.734 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-22T19:51:02.793 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T19:51:02.804 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:51:02.805 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-22T19:51:02.881 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-22T19:51:02.905 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T19:51:02.906 INFO:teuthology.orchestra.run.trial042.stdout:Setting up librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:02.907 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:02.908 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T19:51:02.952 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-22T19:51:03.027 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-22T19:51:03.101 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-22T19:51:03.171 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-22T19:51:03.249 INFO:teuthology.orchestra.run.trial066.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T19:51:03.250 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-22T19:51:03.330 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T19:51:03.331 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T19:51:03.407 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T19:51:03.425 INFO:teuthology.orchestra.run.trial042.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-22T19:51:03.436 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.437 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.437 INFO:teuthology.orchestra.run.trial042.stdout:Setting up librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.438 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.439 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.485 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T19:51:03.508 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T19:51:03.509 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T19:51:03.567 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-22T19:51:03.640 INFO:teuthology.orchestra.run.trial066.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T19:51:03.641 INFO:teuthology.orchestra.run.trial066.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-22T19:51:03.642 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T19:51:03.643 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T19:51:03.778 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-22T19:51:03.855 INFO:teuthology.orchestra.run.trial066.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-22T19:51:03.856 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-22T19:51:03.890 INFO:teuthology.orchestra.run.trial042.stdout:Setting up libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.891 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.892 INFO:teuthology.orchestra.run.trial042.stdout:Setting up librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.893 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.893 INFO:teuthology.orchestra.run.trial042.stdout:Setting up rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.894 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.895 INFO:teuthology.orchestra.run.trial042.stdout:Setting up python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.896 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.925 INFO:teuthology.orchestra.run.trial042.stdout:Adding group ceph....done 2026-02-22T19:51:03.928 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:51:03.929 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-22T19:51:03.960 INFO:teuthology.orchestra.run.trial042.stdout:Adding system user ceph....done 2026-02-22T19:51:03.970 INFO:teuthology.orchestra.run.trial042.stdout:Setting system user ceph properties....done 2026-02-22T19:51:03.976 INFO:teuthology.orchestra.run.trial042.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-22T19:51:04.012 INFO:teuthology.orchestra.run.trial066.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-22T19:51:04.013 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-22T19:51:04.063 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-22T19:51:04.091 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-22T19:51:04.221 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-22T19:51:04.307 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T19:51:04.311 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-22T19:51:04.422 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T19:51:04.423 INFO:teuthology.orchestra.run.trial066.stdout:Setting up librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:04.424 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:04.425 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T19:51:04.660 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:04.661 INFO:teuthology.orchestra.run.trial042.stdout:Setting up radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:04.916 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T19:51:04.916 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T19:51:04.938 INFO:teuthology.orchestra.run.trial066.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-22T19:51:04.949 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:04.950 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:04.951 INFO:teuthology.orchestra.run.trial066.stdout:Setting up librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:04.952 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:04.953 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.021 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T19:51:05.022 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T19:51:05.315 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.421 INFO:teuthology.orchestra.run.trial066.stdout:Setting up libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.422 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.423 INFO:teuthology.orchestra.run.trial066.stdout:Setting up librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.424 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.425 INFO:teuthology.orchestra.run.trial066.stdout:Setting up rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.425 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.426 INFO:teuthology.orchestra.run.trial066.stdout:Setting up python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.427 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.434 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-22T19:51:05.456 INFO:teuthology.orchestra.run.trial066.stdout:Adding group ceph....done 2026-02-22T19:51:05.490 INFO:teuthology.orchestra.run.trial066.stdout:Adding system user ceph....done 2026-02-22T19:51:05.500 INFO:teuthology.orchestra.run.trial066.stdout:Setting system user ceph properties....done 2026-02-22T19:51:05.505 INFO:teuthology.orchestra.run.trial066.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-22T19:51:05.593 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-22T19:51:05.826 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.878 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-22T19:51:05.886 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T19:51:05.886 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T19:51:06.274 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.274 INFO:teuthology.orchestra.run.trial042.stdout:Setting up rbd-mirror (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.275 INFO:teuthology.orchestra.run.trial066.stdout:Setting up radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.350 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T19:51:06.350 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T19:51:06.559 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T19:51:06.560 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T19:51:06.714 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.794 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T19:51:06.794 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T19:51:06.953 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:07.069 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-22T19:51:07.154 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:07.238 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T19:51:07.238 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T19:51:07.458 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:07.517 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T19:51:07.518 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T19:51:07.625 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:07.626 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:07.636 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:07.708 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T19:51:07.708 INFO:teuthology.orchestra.run.trial042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T19:51:07.905 INFO:teuthology.orchestra.run.trial066.stdout:Setting up rbd-mirror (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:07.982 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T19:51:07.982 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T19:51:08.077 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:08.087 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:08.088 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:08.099 INFO:teuthology.orchestra.run.trial042.stdout:Setting up ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:08.208 INFO:teuthology.orchestra.run.trial042.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-22T19:51:08.215 INFO:teuthology.orchestra.run.trial042.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-22T19:51:08.378 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:08.438 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T19:51:08.438 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T19:51:08.454 INFO:teuthology.orchestra.run.trial042.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-22T19:51:08.537 INFO:teuthology.orchestra.run.trial042.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-22T19:51:08.821 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:08.917 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T19:51:08.917 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T19:51:09.019 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:09.019 INFO:teuthology.orchestra.run.trial042.stdout:Running kernel seems to be up-to-date. 2026-02-22T19:51:09.019 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:09.019 INFO:teuthology.orchestra.run.trial042.stdout:The processor microcode seems to be up-to-date. 2026-02-22T19:51:09.019 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:09.019 INFO:teuthology.orchestra.run.trial042.stdout:No services need to be restarted. 2026-02-22T19:51:09.019 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:09.019 INFO:teuthology.orchestra.run.trial042.stdout:No containers need to be restarted. 2026-02-22T19:51:09.020 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:09.020 INFO:teuthology.orchestra.run.trial042.stdout:No user sessions are running outdated binaries. 2026-02-22T19:51:09.020 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:09.020 INFO:teuthology.orchestra.run.trial042.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-22T19:51:09.280 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:09.280 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:09.291 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:09.363 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T19:51:09.363 INFO:teuthology.orchestra.run.trial066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T19:51:09.758 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:09.768 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:09.769 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:09.779 INFO:teuthology.orchestra.run.trial066.stdout:Setting up ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:09.815 INFO:teuthology.orchestra.run.trial042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-22T19:51:09.819 DEBUG:teuthology.parallel:result is None 2026-02-22T19:51:09.887 INFO:teuthology.orchestra.run.trial066.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-22T19:51:09.894 INFO:teuthology.orchestra.run.trial066.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-22T19:51:10.135 INFO:teuthology.orchestra.run.trial066.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-22T19:51:10.156 INFO:teuthology.orchestra.run.trial066.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-22T19:51:10.634 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:10.634 INFO:teuthology.orchestra.run.trial066.stdout:Running kernel seems to be up-to-date. 2026-02-22T19:51:10.634 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:10.634 INFO:teuthology.orchestra.run.trial066.stdout:The processor microcode seems to be up-to-date. 2026-02-22T19:51:10.634 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:10.635 INFO:teuthology.orchestra.run.trial066.stdout:No services need to be restarted. 2026-02-22T19:51:10.635 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:10.635 INFO:teuthology.orchestra.run.trial066.stdout:No containers need to be restarted. 2026-02-22T19:51:10.635 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:10.635 INFO:teuthology.orchestra.run.trial066.stdout:No user sessions are running outdated binaries. 2026-02-22T19:51:10.635 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:10.635 INFO:teuthology.orchestra.run.trial066.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-22T19:51:11.432 INFO:teuthology.orchestra.run.trial066.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-22T19:51:11.438 DEBUG:teuthology.parallel:result is None 2026-02-22T19:51:11.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:51:11.606 DEBUG:teuthology.orchestra.run.trial042:> dpkg-query -W -f '${Version}' ceph 2026-02-22T19:51:11.616 INFO:teuthology.orchestra.run.trial042.stdout:19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:51:11.616 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:51:11.617 INFO:teuthology.task.install:The correct ceph version 19.2.3-689-gf90f4eae-1jammy is installed. 2026-02-22T19:51:11.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:51:11.752 DEBUG:teuthology.orchestra.run.trial066:> dpkg-query -W -f '${Version}' ceph 2026-02-22T19:51:11.762 INFO:teuthology.orchestra.run.trial066.stdout:19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:51:11.763 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:51:11.763 INFO:teuthology.task.install:The correct ceph version 19.2.3-689-gf90f4eae-1jammy is installed. 2026-02-22T19:51:11.766 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-22T19:51:11.766 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:11.766 DEBUG:teuthology.orchestra.run.trial042:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T19:51:11.777 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:11.777 DEBUG:teuthology.orchestra.run.trial066:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T19:51:11.814 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-22T19:51:11.815 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:11.815 DEBUG:teuthology.orchestra.run.trial042:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T19:51:11.828 DEBUG:teuthology.orchestra.run.trial042:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T19:51:11.879 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:11.879 DEBUG:teuthology.orchestra.run.trial066:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T19:51:11.887 DEBUG:teuthology.orchestra.run.trial066:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T19:51:11.938 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-22T19:51:11.938 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:11.938 DEBUG:teuthology.orchestra.run.trial042:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T19:51:11.948 DEBUG:teuthology.orchestra.run.trial042:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T19:51:11.999 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:11.999 DEBUG:teuthology.orchestra.run.trial066:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T19:51:12.007 DEBUG:teuthology.orchestra.run.trial066:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T19:51:12.058 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-22T19:51:12.058 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:12.058 DEBUG:teuthology.orchestra.run.trial042:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T19:51:12.068 DEBUG:teuthology.orchestra.run.trial042:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T19:51:12.123 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:12.123 DEBUG:teuthology.orchestra.run.trial066:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T19:51:12.131 DEBUG:teuthology.orchestra.run.trial066:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T19:51:12.181 INFO:teuthology.run_tasks:Running task ceph... 2026-02-22T19:51:12.256 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-22T19:51:12.257 DEBUG:teuthology.orchestra.run.trial042:> sudo chmod 777 /var/log/ceph 2026-02-22T19:51:12.258 DEBUG:teuthology.orchestra.run.trial066:> sudo chmod 777 /var/log/ceph 2026-02-22T19:51:12.265 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-22T19:51:12.265 DEBUG:teuthology.orchestra.run.trial042:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T19:51:12.308 DEBUG:teuthology.orchestra.run.trial066:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T19:51:12.315 INFO:tasks.ceph:Creating extra log directories... 2026-02-22T19:51:12.315 DEBUG:teuthology.orchestra.run.trial042:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T19:51:12.356 DEBUG:teuthology.orchestra.run.trial066:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T19:51:12.391 INFO:tasks.ceph:Creating ceph cluster cluster1... 2026-02-22T19:51:12.391 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-22T19:51:12.391 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972', 'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '63972', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9196, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'targets': {'trial042.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJFOY7gCzruyFRuLBxtBoDvQgtaYFFQOw5vx3RBuU1mB/Ah/xwnZFQzE4e9LqJjI8XDxcQI6v5hZ7Xf3q0Mv6lg=', 'trial066.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdYkUMhYRZ+Fncb1nPoj2MOd3bMQdmVaMHKfRqH7QQahTfDkUNFuUFQ6ruWGi0llCxilrcV5oFjpJ06/7W/PBc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'pool', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'journal', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-22_19:35:38', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-22T19:51:12.392 DEBUG:teuthology.orchestra.run.trial042:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2026-02-22T19:51:12.404 DEBUG:teuthology.orchestra.run.trial066:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2026-02-22T19:51:12.414 DEBUG:teuthology.orchestra.run.trial042:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T19:51:12.448 DEBUG:teuthology.orchestra.run.trial066:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T19:51:12.462 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:12.462 DEBUG:teuthology.orchestra.run.trial042:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T19:51:12.502 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T19:51:12.503 DEBUG:teuthology.orchestra.run.trial042:> stat /dev/vg_nvme/lv_1 2026-02-22T19:51:12.546 INFO:teuthology.orchestra.run.trial042.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T19:51:12.546 INFO:teuthology.orchestra.run.trial042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:12.546 INFO:teuthology.orchestra.run.trial042.stdout:Device: 5h/5d Inode: 907 Links: 1 2026-02-22T19:51:12.546 INFO:teuthology.orchestra.run.trial042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:12.546 INFO:teuthology.orchestra.run.trial042.stdout:Access: 2026-02-22 19:46:43.723661148 +0000 2026-02-22T19:51:12.546 INFO:teuthology.orchestra.run.trial042.stdout:Modify: 2026-02-22 19:46:43.563663875 +0000 2026-02-22T19:51:12.546 INFO:teuthology.orchestra.run.trial042.stdout:Change: 2026-02-22 19:46:43.563663875 +0000 2026-02-22T19:51:12.546 INFO:teuthology.orchestra.run.trial042.stdout: Birth: - 2026-02-22T19:51:12.546 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T19:51:12.594 INFO:teuthology.orchestra.run.trial042.stderr:1+0 records in 2026-02-22T19:51:12.594 INFO:teuthology.orchestra.run.trial042.stderr:1+0 records out 2026-02-22T19:51:12.594 INFO:teuthology.orchestra.run.trial042.stderr:512 bytes copied, 0.000182701 s, 2.8 MB/s 2026-02-22T19:51:12.595 DEBUG:teuthology.orchestra.run.trial042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T19:51:12.640 DEBUG:teuthology.orchestra.run.trial042:> stat /dev/vg_nvme/lv_2 2026-02-22T19:51:12.686 INFO:teuthology.orchestra.run.trial042.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T19:51:12.687 INFO:teuthology.orchestra.run.trial042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:12.687 INFO:teuthology.orchestra.run.trial042.stdout:Device: 5h/5d Inode: 942 Links: 1 2026-02-22T19:51:12.687 INFO:teuthology.orchestra.run.trial042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:12.687 INFO:teuthology.orchestra.run.trial042.stdout:Access: 2026-02-22 19:46:43.883658424 +0000 2026-02-22T19:51:12.687 INFO:teuthology.orchestra.run.trial042.stdout:Modify: 2026-02-22 19:46:43.879658493 +0000 2026-02-22T19:51:12.687 INFO:teuthology.orchestra.run.trial042.stdout:Change: 2026-02-22 19:46:43.879658493 +0000 2026-02-22T19:51:12.687 INFO:teuthology.orchestra.run.trial042.stdout: Birth: - 2026-02-22T19:51:12.687 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T19:51:12.738 INFO:teuthology.orchestra.run.trial042.stderr:1+0 records in 2026-02-22T19:51:12.738 INFO:teuthology.orchestra.run.trial042.stderr:1+0 records out 2026-02-22T19:51:12.738 INFO:teuthology.orchestra.run.trial042.stderr:512 bytes copied, 0.000179605 s, 2.9 MB/s 2026-02-22T19:51:12.739 DEBUG:teuthology.orchestra.run.trial042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T19:51:12.788 DEBUG:teuthology.orchestra.run.trial042:> stat /dev/vg_nvme/lv_3 2026-02-22T19:51:12.834 INFO:teuthology.orchestra.run.trial042.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T19:51:12.834 INFO:teuthology.orchestra.run.trial042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:12.835 INFO:teuthology.orchestra.run.trial042.stdout:Device: 5h/5d Inode: 954 Links: 1 2026-02-22T19:51:12.835 INFO:teuthology.orchestra.run.trial042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:12.835 INFO:teuthology.orchestra.run.trial042.stdout:Access: 2026-02-22 19:46:44.355650386 +0000 2026-02-22T19:51:12.835 INFO:teuthology.orchestra.run.trial042.stdout:Modify: 2026-02-22 19:46:44.187653246 +0000 2026-02-22T19:51:12.835 INFO:teuthology.orchestra.run.trial042.stdout:Change: 2026-02-22 19:46:44.187653246 +0000 2026-02-22T19:51:12.835 INFO:teuthology.orchestra.run.trial042.stdout: Birth: - 2026-02-22T19:51:12.835 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T19:51:12.881 INFO:teuthology.orchestra.run.trial042.stderr:1+0 records in 2026-02-22T19:51:12.881 INFO:teuthology.orchestra.run.trial042.stderr:1+0 records out 2026-02-22T19:51:12.881 INFO:teuthology.orchestra.run.trial042.stderr:512 bytes copied, 0.000202358 s, 2.5 MB/s 2026-02-22T19:51:12.882 DEBUG:teuthology.orchestra.run.trial042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T19:51:12.928 DEBUG:teuthology.orchestra.run.trial042:> stat /dev/vg_nvme/lv_4 2026-02-22T19:51:12.975 INFO:teuthology.orchestra.run.trial042.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T19:51:12.975 INFO:teuthology.orchestra.run.trial042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:12.975 INFO:teuthology.orchestra.run.trial042.stdout:Device: 5h/5d Inode: 978 Links: 1 2026-02-22T19:51:12.975 INFO:teuthology.orchestra.run.trial042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:12.975 INFO:teuthology.orchestra.run.trial042.stdout:Access: 2026-02-22 19:46:44.679644868 +0000 2026-02-22T19:51:12.975 INFO:teuthology.orchestra.run.trial042.stdout:Modify: 2026-02-22 19:46:44.503647866 +0000 2026-02-22T19:51:12.975 INFO:teuthology.orchestra.run.trial042.stdout:Change: 2026-02-22 19:46:44.503647866 +0000 2026-02-22T19:51:12.975 INFO:teuthology.orchestra.run.trial042.stdout: Birth: - 2026-02-22T19:51:12.976 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T19:51:13.021 INFO:teuthology.orchestra.run.trial042.stderr:1+0 records in 2026-02-22T19:51:13.022 INFO:teuthology.orchestra.run.trial042.stderr:1+0 records out 2026-02-22T19:51:13.022 INFO:teuthology.orchestra.run.trial042.stderr:512 bytes copied, 0.000174054 s, 2.9 MB/s 2026-02-22T19:51:13.023 DEBUG:teuthology.orchestra.run.trial042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T19:51:13.072 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:13.072 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial042.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T19:51:13.072 INFO:tasks.ceph:Generating config... 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client] debug rbd = 20 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client] log to stderr = False 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client] rbd default features = 125 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.074 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.074 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-22T19:51:13.074 INFO:tasks.ceph:[global] mon warn on pool no app = False 2026-02-22T19:51:13.074 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-22T19:51:13.074 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-22T19:51:13.074 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-22T19:51:13.074 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-22T19:51:13.074 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-22T19:51:13.074 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-22T19:51:13.074 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-22T19:51:13.074 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-22T19:51:13.074 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2026-02-22T19:51:13.074 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-22T19:51:13.075 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-22T19:51:13.075 INFO:tasks.ceph:Setting up cluster1.mon.a... 2026-02-22T19:51:13.075 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2026-02-22T19:51:13.131 INFO:teuthology.orchestra.run.trial042.stdout:creating /etc/ceph/cluster1.keyring 2026-02-22T19:51:13.134 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2026-02-22T19:51:13.197 DEBUG:teuthology.orchestra.run.trial042:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-22T19:51:13.251 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '10.20.193.42')] 2026-02-22T19:51:13.252 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.42', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2026-02-22T19:51:13.253 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:13.253 DEBUG:teuthology.orchestra.run.trial042:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T19:51:13.295 DEBUG:teuthology.orchestra.run.trial042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.42 --print /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:13.351 INFO:teuthology.orchestra.run.trial042.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:13.351 INFO:teuthology.orchestra.run.trial042.stdout:monmaptool: generated fsid de51f049-31b3-4825-9b8c-01cc5ba58723 2026-02-22T19:51:13.351 INFO:teuthology.orchestra.run.trial042.stdout:setting min_mon_release = quincy 2026-02-22T19:51:13.351 INFO:teuthology.orchestra.run.trial042.stdout:epoch 0 2026-02-22T19:51:13.351 INFO:teuthology.orchestra.run.trial042.stdout:fsid de51f049-31b3-4825-9b8c-01cc5ba58723 2026-02-22T19:51:13.351 INFO:teuthology.orchestra.run.trial042.stdout:last_changed 2026-02-22T19:51:13.352022+0000 2026-02-22T19:51:13.351 INFO:teuthology.orchestra.run.trial042.stdout:created 2026-02-22T19:51:13.352022+0000 2026-02-22T19:51:13.352 INFO:teuthology.orchestra.run.trial042.stdout:min_mon_release 17 (quincy) 2026-02-22T19:51:13.352 INFO:teuthology.orchestra.run.trial042.stdout:election_strategy: 1 2026-02-22T19:51:13.352 INFO:teuthology.orchestra.run.trial042.stdout:0: [v2:10.20.193.42:3300/0,v1:10.20.193.42:6789/0] mon.a 2026-02-22T19:51:13.352 INFO:teuthology.orchestra.run.trial042.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2026-02-22T19:51:13.352 DEBUG:teuthology.orchestra.run.trial042:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T19:51:13.398 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID de51f049-31b3-4825-9b8c-01cc5ba58723... 2026-02-22T19:51:13.399 DEBUG:teuthology.orchestra.run.trial042:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2026-02-22T19:51:13.444 DEBUG:teuthology.orchestra.run.trial066:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2026-02-22T19:51:13.462 INFO:teuthology.orchestra.run.trial042.stdout:[global] 2026-02-22T19:51:13.463 INFO:teuthology.orchestra.run.trial042.stdout: chdir = "" 2026-02-22T19:51:13.463 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T19:51:13.463 INFO:teuthology.orchestra.run.trial042.stdout: auth supported = cephx 2026-02-22T19:51:13.463 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.463 INFO:teuthology.orchestra.run.trial042.stdout: filestore xattr use omap = true 2026-02-22T19:51:13.463 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.463 INFO:teuthology.orchestra.run.trial042.stdout: mon clock drift allowed = 1.000 2026-02-22T19:51:13.463 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.463 INFO:teuthology.orchestra.run.trial042.stdout: osd crush chooseleaf type = 0 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: auth debug = true 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: ms die on old message = true 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: ms die on bug = true 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: mon pg warn max object skew = 0 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.464 INFO:teuthology.orchestra.run.trial042.stdout: osd pool default size = 2 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon osd allow primary affinity = true 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon osd allow pg remap = true 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on legacy crush tunables = false 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on crush straw calc version zero = false 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on no sortbitwise = false 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on osd down out interval zero = false 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on too few osds = false 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: mon_allow_pool_size_one = true 2026-02-22T19:51:13.465 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: osd default data pool replay window = 5 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: mon allow pool delete = true 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: mon cluster log file level = debug 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: debug asserts on shutdown = true 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: mon health detail to clog = false 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: mon host = 10.20.193.42 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: mon client directed command retry = 5 2026-02-22T19:51:13.466 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on pool no app = False 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: ms inject socket failures = 5000 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: fsid = de51f049-31b3-4825-9b8c-01cc5ba58723 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout:[osd] 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: osd journal size = 100 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: osd scrub load threshold = 5.0 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: osd scrub max interval = 600 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: osd mclock profile = high_recovery_ops 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: osd mclock skip benchmark = true 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.467 INFO:teuthology.orchestra.run.trial042.stdout: osd recover clone overlap = true 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: osd recovery max chunk = 1048576 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: osd debug shutdown = true 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: osd debug op order = true 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: osd debug verify stray on activate = true 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: osd debug trim objects = true 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: osd open classes on start = true 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: osd debug pg log writeout = true 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: osd deep scrub update digest min age = 30 2026-02-22T19:51:13.468 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: osd map max advance = 10 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: journal zero on create = true 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: filestore ondisk finisher threads = 3 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: filestore apply finisher threads = 3 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: bdev debug aio = true 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: osd debug misdirected ops = true 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: bdev async discard = True 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: bdev enable discard = True 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: bluestore allocator = stupid 2026-02-22T19:51:13.469 INFO:teuthology.orchestra.run.trial042.stdout: bluestore block size = 96636764160 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: bluestore fsck on mount = True 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: debug bluefs = 1/20 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: debug bluestore = 1/20 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: debug ms = 1 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: debug osd = 20 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: debug rocksdb = 4/10 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: mon osd full ratio = 0.9 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: mon osd nearfull ratio = 0.8 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: osd failsafe full ratio = 0.95 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: osd objectstore = bluestore 2026-02-22T19:51:13.470 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout:[mgr] 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: debug ms = 1 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: debug mgr = 20 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: debug mon = 20 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: debug auth = 20 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: mgr/telemetry/nag = false 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout:[mon] 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: debug ms = 1 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: debug mon = 20 2026-02-22T19:51:13.471 INFO:teuthology.orchestra.run.trial042.stdout: debug paxos = 20 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: debug auth = 20 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: mon data avail warn = 5 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: mon mgr mkfs grace = 240 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: mon osd reporter subtree level = osd 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: mon osd prime pg temp = true 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T19:51:13.472 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: # 1m isn't quite enough 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: mon_down_mkfs_grace = 2m 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_filestore_osds = false 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout:[client] 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: rgw cache enabled = true 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: rgw enable ops log = true 2026-02-22T19:51:13.473 INFO:teuthology.orchestra.run.trial042.stdout: rgw enable usage log = true 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout: debug rbd = 20 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout: debug rbd_mirror = 15 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout: log to stderr = False 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout: rbd default features = 125 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout: rbd mirror image policy type = simple 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout:[mon.a] 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.0] 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.474 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.1] 2026-02-22T19:51:13.475 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.475 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.475 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.2] 2026-02-22T19:51:13.475 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.475 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.475 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.3] 2026-02-22T19:51:13.475 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.475 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.477 INFO:teuthology.orchestra.run.trial066.stdout:[global] 2026-02-22T19:51:13.477 INFO:teuthology.orchestra.run.trial066.stdout: chdir = "" 2026-02-22T19:51:13.477 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T19:51:13.477 INFO:teuthology.orchestra.run.trial066.stdout: auth supported = cephx 2026-02-22T19:51:13.477 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.477 INFO:teuthology.orchestra.run.trial066.stdout: filestore xattr use omap = true 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: mon clock drift allowed = 1.000 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: osd crush chooseleaf type = 0 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: auth debug = true 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: ms die on old message = true 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: ms die on bug = true 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: mon pg warn max object skew = 0 2026-02-22T19:51:13.478 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: osd pool default size = 2 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: mon osd allow primary affinity = true 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: mon osd allow pg remap = true 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on legacy crush tunables = false 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on crush straw calc version zero = false 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on no sortbitwise = false 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on osd down out interval zero = false 2026-02-22T19:51:13.479 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on too few osds = false 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: mon_allow_pool_size_one = true 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: osd default data pool replay window = 5 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: mon allow pool delete = true 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: mon cluster log file level = debug 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: debug asserts on shutdown = true 2026-02-22T19:51:13.480 INFO:teuthology.orchestra.run.trial066.stdout: mon health detail to clog = false 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: mon host = 10.20.193.42 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: mon client directed command retry = 5 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on pool no app = False 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: ms inject socket failures = 5000 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: fsid = de51f049-31b3-4825-9b8c-01cc5ba58723 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout:[osd] 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: osd journal size = 100 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: osd scrub load threshold = 5.0 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: osd scrub max interval = 600 2026-02-22T19:51:13.481 INFO:teuthology.orchestra.run.trial066.stdout: osd mclock profile = high_recovery_ops 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: osd mclock skip benchmark = true 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: osd recover clone overlap = true 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: osd recovery max chunk = 1048576 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: osd debug shutdown = true 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: osd debug op order = true 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: osd debug verify stray on activate = true 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: osd debug trim objects = true 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: osd open classes on start = true 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: osd debug pg log writeout = true 2026-02-22T19:51:13.482 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: osd deep scrub update digest min age = 30 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: osd map max advance = 10 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: journal zero on create = true 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: filestore ondisk finisher threads = 3 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: filestore apply finisher threads = 3 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: bdev debug aio = true 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: osd debug misdirected ops = true 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: bdev async discard = True 2026-02-22T19:51:13.483 INFO:teuthology.orchestra.run.trial066.stdout: bdev enable discard = True 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: bluestore allocator = stupid 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: bluestore block size = 96636764160 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: bluestore fsck on mount = True 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: debug bluefs = 1/20 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: debug bluestore = 1/20 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: debug ms = 1 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: debug osd = 20 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: debug rocksdb = 4/10 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: mon osd full ratio = 0.9 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: mon osd nearfull ratio = 0.8 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: osd failsafe full ratio = 0.95 2026-02-22T19:51:13.484 INFO:teuthology.orchestra.run.trial066.stdout: osd objectstore = bluestore 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout:[mgr] 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: debug ms = 1 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: debug mgr = 20 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: debug mon = 20 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: debug auth = 20 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: mgr/telemetry/nag = false 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout:[mon] 2026-02-22T19:51:13.485 INFO:teuthology.orchestra.run.trial066.stdout: debug ms = 1 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: debug mon = 20 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: debug paxos = 20 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: debug auth = 20 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: mon data avail warn = 5 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: mon mgr mkfs grace = 240 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: mon osd reporter subtree level = osd 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: mon osd prime pg temp = true 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T19:51:13.486 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: # 1m isn't quite enough 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: mon_down_mkfs_grace = 2m 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_filestore_osds = false 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout:[client] 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: rgw cache enabled = true 2026-02-22T19:51:13.487 INFO:teuthology.orchestra.run.trial066.stdout: rgw enable ops log = true 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: rgw enable usage log = true 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: debug rbd = 20 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: debug rbd_mirror = 15 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: log to stderr = False 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: rbd default features = 125 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: rbd mirror image policy type = simple 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout:[mon.a] 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.0] 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.488 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.1] 2026-02-22T19:51:13.489 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.489 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.489 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.2] 2026-02-22T19:51:13.489 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.489 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.489 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.3] 2026-02-22T19:51:13.489 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:13.489 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:13.489 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2026-02-22T19:51:13.490 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2026-02-22T19:51:13.541 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-22T19:51:13.541 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:13.541 DEBUG:teuthology.orchestra.run.trial042:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2026-02-22T19:51:13.587 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:13.587 DEBUG:teuthology.orchestra.run.trial042:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2026-02-22T19:51:13.635 INFO:tasks.ceph:Sending monmap to node ubuntu@trial042.front.sepia.ceph.com 2026-02-22T19:51:13.635 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:13.635 DEBUG:teuthology.orchestra.run.trial042:> sudo dd of=/etc/ceph/cluster1.keyring 2026-02-22T19:51:13.635 DEBUG:teuthology.orchestra.run.trial042:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-22T19:51:13.688 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:13.688 DEBUG:teuthology.orchestra.run.trial042:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:13.735 INFO:tasks.ceph:Sending monmap to node ubuntu@trial066.front.sepia.ceph.com 2026-02-22T19:51:13.735 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:13.735 DEBUG:teuthology.orchestra.run.trial066:> sudo dd of=/etc/ceph/cluster1.keyring 2026-02-22T19:51:13.736 DEBUG:teuthology.orchestra.run.trial066:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-22T19:51:13.751 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:13.752 DEBUG:teuthology.orchestra.run.trial066:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:13.798 INFO:tasks.ceph:Setting up mon nodes... 2026-02-22T19:51:13.799 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-22T19:51:13.799 DEBUG:teuthology.orchestra.run.trial042:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2026-02-22T19:51:13.821 INFO:teuthology.orchestra.run.trial042.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2026-02-22T19:51:13.823 INFO:tasks.ceph:Setting up mds nodes... 2026-02-22T19:51:13.824 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-22T19:51:13.824 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2026-02-22T19:51:13.883 INFO:teuthology.orchestra.run.trial042.stdout:creating /etc/ceph/cluster1.client.0.keyring 2026-02-22T19:51:13.891 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2026-02-22T19:51:13.910 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2026-02-22T19:51:13.918 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2026-02-22T19:51:13.978 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2026-02-22T19:51:13.985 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2026-02-22T19:51:14.046 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2026-02-22T19:51:14.054 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2026-02-22T19:51:14.114 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2026-02-22T19:51:14.122 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2026-02-22T19:51:14.182 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2026-02-22T19:51:14.190 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2026-02-22T19:51:14.250 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2026-02-22T19:51:14.258 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2026-02-22T19:51:14.318 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2026-02-22T19:51:14.325 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2026-02-22T19:51:14.386 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2026-02-22T19:51:14.393 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-22T19:51:14.393 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial042.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T19:51:14.394 DEBUG:teuthology.orchestra.run.trial042:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2026-02-22T19:51:14.402 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:14.402 INFO:tasks.ceph:role: cluster1.osd.0 2026-02-22T19:51:14.402 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial042.front.sepia.ceph.com 2026-02-22T19:51:14.402 DEBUG:teuthology.orchestra.run.trial042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T19:51:14.452 INFO:teuthology.orchestra.run.trial042.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:14.452 INFO:teuthology.orchestra.run.trial042.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:14.452 INFO:teuthology.orchestra.run.trial042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:14.452 INFO:teuthology.orchestra.run.trial042.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:14.453 INFO:teuthology.orchestra.run.trial042.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:14.453 INFO:teuthology.orchestra.run.trial042.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:14.453 INFO:teuthology.orchestra.run.trial042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:14.453 INFO:teuthology.orchestra.run.trial042.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:14.453 INFO:teuthology.orchestra.run.trial042.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:14.453 INFO:teuthology.orchestra.run.trial042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:15.046 INFO:teuthology.orchestra.run.trial042.stdout:Discarding blocks...Done. 2026-02-22T19:51:15.048 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial042.front.sepia.ceph.com -o noatime 2026-02-22T19:51:15.048 DEBUG:teuthology.orchestra.run.trial042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2026-02-22T19:51:15.103 DEBUG:teuthology.orchestra.run.trial042:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2026-02-22T19:51:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:15.153 INFO:teuthology.orchestra.run.trial042.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:15.153 DEBUG:teuthology.orchestra.run.trial042:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2026-02-22T19:51:15.202 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:15.202 INFO:tasks.ceph:role: cluster1.osd.1 2026-02-22T19:51:15.202 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial042.front.sepia.ceph.com 2026-02-22T19:51:15.203 DEBUG:teuthology.orchestra.run.trial042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T19:51:15.253 INFO:teuthology.orchestra.run.trial042.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:15.253 INFO:teuthology.orchestra.run.trial042.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:15.253 INFO:teuthology.orchestra.run.trial042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:15.253 INFO:teuthology.orchestra.run.trial042.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:15.253 INFO:teuthology.orchestra.run.trial042.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:15.253 INFO:teuthology.orchestra.run.trial042.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:15.253 INFO:teuthology.orchestra.run.trial042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:15.254 INFO:teuthology.orchestra.run.trial042.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:15.254 INFO:teuthology.orchestra.run.trial042.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:15.254 INFO:teuthology.orchestra.run.trial042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:15.817 INFO:teuthology.orchestra.run.trial042.stdout:Discarding blocks...Done. 2026-02-22T19:51:15.819 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial042.front.sepia.ceph.com -o noatime 2026-02-22T19:51:15.819 DEBUG:teuthology.orchestra.run.trial042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2026-02-22T19:51:15.872 DEBUG:teuthology.orchestra.run.trial042:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2026-02-22T19:51:15.921 INFO:teuthology.orchestra.run.trial042.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:15.922 DEBUG:teuthology.orchestra.run.trial042:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2026-02-22T19:51:15.971 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:15.971 INFO:tasks.ceph:role: cluster1.osd.2 2026-02-22T19:51:15.971 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial042.front.sepia.ceph.com 2026-02-22T19:51:15.972 DEBUG:teuthology.orchestra.run.trial042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T19:51:16.025 INFO:teuthology.orchestra.run.trial042.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:16.025 INFO:teuthology.orchestra.run.trial042.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:16.025 INFO:teuthology.orchestra.run.trial042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:16.025 INFO:teuthology.orchestra.run.trial042.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:16.025 INFO:teuthology.orchestra.run.trial042.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:16.025 INFO:teuthology.orchestra.run.trial042.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:16.025 INFO:teuthology.orchestra.run.trial042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:16.025 INFO:teuthology.orchestra.run.trial042.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:16.025 INFO:teuthology.orchestra.run.trial042.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:16.026 INFO:teuthology.orchestra.run.trial042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:16.601 INFO:teuthology.orchestra.run.trial042.stdout:Discarding blocks...Done. 2026-02-22T19:51:16.602 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial042.front.sepia.ceph.com -o noatime 2026-02-22T19:51:16.603 DEBUG:teuthology.orchestra.run.trial042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2026-02-22T19:51:16.656 DEBUG:teuthology.orchestra.run.trial042:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2026-02-22T19:51:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:16.706 INFO:teuthology.orchestra.run.trial042.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:16.706 DEBUG:teuthology.orchestra.run.trial042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:16.767 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:16.765+0000 7f0027e598c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2026-02-22T19:51:16.768 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:16.765+0000 7f0027e598c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2026-02-22T19:51:16.768 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:16.765+0000 7f0027e598c0 -1 bdev(0x5593d73c1800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:16.768 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:16.765+0000 7f0027e598c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2026-02-22T19:51:20.883 DEBUG:teuthology.orchestra.run.trial042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2026-02-22T19:51:20.936 DEBUG:teuthology.orchestra.run.trial042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:21.000 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:20.997+0000 7f9b268748c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2026-02-22T19:51:21.000 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:20.997+0000 7f9b268748c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2026-02-22T19:51:21.000 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:20.997+0000 7f9b268748c0 -1 bdev(0x56039b2df800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:21.000 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:20.997+0000 7f9b268748c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2026-02-22T19:51:25.070 DEBUG:teuthology.orchestra.run.trial042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2026-02-22T19:51:25.120 DEBUG:teuthology.orchestra.run.trial042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:25.184 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:25.181+0000 7f747c8b08c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2026-02-22T19:51:25.184 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:25.181+0000 7f747c8b08c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2026-02-22T19:51:25.184 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:25.181+0000 7f747c8b08c0 -1 bdev(0x562fc790d800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:25.185 INFO:teuthology.orchestra.run.trial042.stderr:2026-02-22T19:51:25.181+0000 7f747c8b08c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2026-02-22T19:51:29.254 DEBUG:teuthology.orchestra.run.trial042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2026-02-22T19:51:29.308 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-22T19:51:29.309 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:29.309 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2026-02-22T19:51:29.360 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:29.360 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2026-02-22T19:51:29.412 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:29.412 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2026-02-22T19:51:29.464 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:29.464 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2026-02-22T19:51:29.516 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:29.516 DEBUG:teuthology.orchestra.run.trial042:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2026-02-22T19:51:29.563 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:29.564 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2026-02-22T19:51:29.569 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:29.570 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2026-02-22T19:51:29.614 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:29.614 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2026-02-22T19:51:29.662 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:29.662 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2026-02-22T19:51:29.711 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:29.711 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2026-02-22T19:51:29.759 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:29.759 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2026-02-22T19:51:29.806 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:29.806 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2026-02-22T19:51:29.855 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:29.855 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2026-02-22T19:51:29.903 INFO:tasks.ceph:Adding keys to all mons... 2026-02-22T19:51:29.903 DEBUG:teuthology.orchestra.run.trial042:> sudo tee -a /etc/ceph/cluster1.keyring 2026-02-22T19:51:29.912 INFO:teuthology.orchestra.run.trial042.stdout:[mgr.x] 2026-02-22T19:51:29.912 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAxXptpGn36MBAAHCdm9CATxZUP57NxRvMIEg== 2026-02-22T19:51:29.912 INFO:teuthology.orchestra.run.trial042.stdout:[osd.0] 2026-02-22T19:51:29.912 INFO:teuthology.orchestra.run.trial042.stdout: key = AQA0XptprVfHLRAA4kNtFyppEY8BWaQKCg4A/g== 2026-02-22T19:51:29.912 INFO:teuthology.orchestra.run.trial042.stdout:[osd.1] 2026-02-22T19:51:29.912 INFO:teuthology.orchestra.run.trial042.stdout: key = AQA4XptpZMWZOxAAbM/OWqFtRUjKPCwaPvKowQ== 2026-02-22T19:51:29.912 INFO:teuthology.orchestra.run.trial042.stdout:[osd.2] 2026-02-22T19:51:29.912 INFO:teuthology.orchestra.run.trial042.stdout: key = AQA9XptphCn+ChAAAvk+hc88ZbPESUxafGCJhg== 2026-02-22T19:51:29.912 INFO:teuthology.orchestra.run.trial042.stdout:[client.0] 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAxXptp0ty1NBAANnXagMK7l2mmSGpKxQzizw== 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror] 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAxXptpTshCNhAAhfa9tJuc9suH/xoNvQd9ew== 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.0] 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAxXptpYX9NOhAAzPzMz6OaTMBxtjxYrgycaQ== 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.1] 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAyXptp8JTBAhAAbEpGMGHT49SrjPyn7uGqoQ== 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.2] 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAyXptpGnLMBhAA86rh9ZOVFI7aL0cJK6ayXA== 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.3] 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAyXptpRELfChAAV6/ogsPCZr0E+Ta8UBEp8w== 2026-02-22T19:51:29.913 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.4] 2026-02-22T19:51:29.914 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAyXptpRSjrDhAAGlD6Z76PR5DJm1UpDFldiw== 2026-02-22T19:51:29.914 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.5] 2026-02-22T19:51:29.914 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAyXptpCXj2EhAAf0pTdWqLwS9k+3hj6Zjn4Q== 2026-02-22T19:51:29.914 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.6] 2026-02-22T19:51:29.914 INFO:teuthology.orchestra.run.trial042.stdout: key = AQAyXptpwroBFxAAlRr7qL+OlbcY11xWywueKA== 2026-02-22T19:51:29.915 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T19:51:29.980 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:51:30.044 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:51:30.108 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:51:30.171 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:30.236 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:30.300 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:30.364 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:30.428 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:30.492 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:30.556 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:30.620 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:30.685 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:30.748 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-22T19:51:30.748 DEBUG:teuthology.orchestra.run.trial042:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2026-02-22T19:51:30.799 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2026-02-22T19:51:30.864 DEBUG:teuthology.orchestra.run.trial042:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2026-02-22T19:51:30.915 DEBUG:teuthology.orchestra.run.trial042:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:30.962 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2026-02-22T19:51:30.963 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2026-02-22T19:51:30.963 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2026-02-22T19:51:31.004 INFO:tasks.ceph.cluster1.mon.a:Started 2026-02-22T19:51:31.004 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2026-02-22T19:51:31.005 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2026-02-22T19:51:31.005 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2026-02-22T19:51:31.007 INFO:tasks.ceph.cluster1.mgr.x:Started 2026-02-22T19:51:31.007 INFO:tasks.ceph:Setting crush tunables to default 2026-02-22T19:51:31.007 DEBUG:teuthology.orchestra.run.trial042:> sudo ceph --cluster cluster1 osd crush tunables default 2026-02-22T19:51:31.129 INFO:teuthology.orchestra.run.trial042.stderr:adjusted tunables profile to default 2026-02-22T19:51:31.139 INFO:tasks.ceph:check_enable_crimson: False 2026-02-22T19:51:31.139 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2026-02-22T19:51:31.140 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:31.140 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2026-02-22T19:51:31.150 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:31.151 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2026-02-22T19:51:31.200 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:31.200 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2026-02-22T19:51:31.254 DEBUG:teuthology.orchestra.run.trial042:> sudo ceph --cluster cluster1 osd new 89d5fed7-52dc-4e33-8e1e-4c3ec1b58f68 0 2026-02-22T19:51:31.331 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:31.328+0000 7fc2ae09d280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T19:51:31.357 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:31.352+0000 7fc2ae09d280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T19:51:31.394 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:31.388+0000 7fc2ae09d280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T19:51:31.399 INFO:teuthology.orchestra.run.trial042.stdout:0 2026-02-22T19:51:31.411 DEBUG:teuthology.orchestra.run.trial042:> sudo ceph --cluster cluster1 osd new 3dff05e0-678e-4469-8c56-b41daeb8a87a 1 2026-02-22T19:51:31.470 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:31.464+0000 7fc2ae09d280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T19:51:31.546 INFO:teuthology.orchestra.run.trial042.stdout:1 2026-02-22T19:51:31.554 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:31.548+0000 7fc2ae09d280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T19:51:31.556 DEBUG:teuthology.orchestra.run.trial042:> sudo ceph --cluster cluster1 osd new eafdcdb0-26b9-4071-80a8-29486388a96e 2 2026-02-22T19:51:31.593 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:31.588+0000 7fc2ae09d280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T19:51:31.669 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:31.664+0000 7fc2ae09d280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T19:51:31.690 INFO:teuthology.orchestra.run.trial042.stdout:2 2026-02-22T19:51:31.700 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2026-02-22T19:51:31.700 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2026-02-22T19:51:31.702 INFO:tasks.ceph.cluster1.osd.0:Started 2026-02-22T19:51:31.702 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2026-02-22T19:51:31.702 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2026-02-22T19:51:31.704 INFO:tasks.ceph.cluster1.osd.1:Started 2026-02-22T19:51:31.704 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2026-02-22T19:51:31.704 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2026-02-22T19:51:31.706 INFO:tasks.ceph.cluster1.osd.2:Started 2026-02-22T19:51:31.707 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:31.810 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:31.810 INFO:teuthology.orchestra.run.trial042.stdout:{"epoch":5,"fsid":"de51f049-31b3-4825-9b8c-01cc5ba58723","created":"2026-02-22T19:51:31.088879+0000","modified":"2026-02-22T19:51:31.690292+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:51:31.690292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"89d5fed7-52dc-4e33-8e1e-4c3ec1b58f68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dff05e0-678e-4469-8c56-b41daeb8a87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eafdcdb0-26b9-4071-80a8-29486388a96e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:31.821 INFO:tasks.ceph.ceph_manager.cluster1:[] 2026-02-22T19:51:31.822 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-22T19:51:31.851 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:31.848+0000 7fc2ae09d280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T19:51:32.124 DEBUG:teuthology.orchestra.run.trial042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:32.149 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.144+0000 7fc2ae09d280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T19:51:32.211 INFO:teuthology.misc.health.trial042.stdout: 2026-02-22T19:51:32.211 INFO:teuthology.misc.health.trial042.stdout:{"epoch":5,"fsid":"de51f049-31b3-4825-9b8c-01cc5ba58723","created":"2026-02-22T19:51:31.088879+0000","modified":"2026-02-22T19:51:31.690292+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:51:31.690292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"89d5fed7-52dc-4e33-8e1e-4c3ec1b58f68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dff05e0-678e-4469-8c56-b41daeb8a87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eafdcdb0-26b9-4071-80a8-29486388a96e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:32.219 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T19:51:32.241 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.236+0000 7fc2ae09d280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T19:51:32.269 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.264+0000 7fc2ae09d280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T19:51:32.301 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.296+0000 7fc2ae09d280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T19:51:32.344 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.340+0000 7fc2ae09d280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T19:51:32.444 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.440+0000 7fc2ae09d280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T19:51:32.634 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-22T19:51:32.807 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.800+0000 7fc2ae09d280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T19:51:32.838 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.832+0000 7fc2ae09d280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T19:51:32.930 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.924+0000 7fc2ae09d280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T19:51:32.955 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.948+0000 7fc2ae09d280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T19:51:32.981 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:32.976+0000 7fc2ae09d280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T19:51:33.008 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:33.004+0000 7fc2ae09d280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T19:51:33.129 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T19:51:33.129 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T19:51:33.129 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr: from numpy import show_config as show_numpy_config 2026-02-22T19:51:33.135 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:33.132+0000 7fc2ae09d280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T19:51:33.162 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:33.156+0000 7fc2ae09d280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T19:51:33.200 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:33.196+0000 7fc2ae09d280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T19:51:33.528 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T19:51:33.524+0000 7f65fac3e8c0 -1 Falling back to public interface 2026-02-22T19:51:33.732 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T19:51:33.728+0000 7f8db7d768c0 -1 Falling back to public interface 2026-02-22T19:51:33.780 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T19:51:33.776+0000 7f5e1ab978c0 -1 Falling back to public interface 2026-02-22T19:51:34.226 INFO:tasks.ceph.cluster1.mgr.x.trial042.stderr:2026-02-22T19:51:34.220+0000 7fc1cc7e9640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-22T19:51:38.385 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T19:51:38.380+0000 7f65fac3e8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T19:51:38.504 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T19:51:38.500+0000 7f8db7d768c0 -1 osd.1 0 log_to_monitors true 2026-02-22T19:51:38.523 DEBUG:teuthology.orchestra.run.trial042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:38.640 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T19:51:38.636+0000 7f5e1ab978c0 -1 osd.2 0 log_to_monitors true 2026-02-22T19:51:38.673 INFO:teuthology.misc.health.trial042.stdout: 2026-02-22T19:51:38.673 INFO:teuthology.misc.health.trial042.stdout:{"epoch":6,"fsid":"de51f049-31b3-4825-9b8c-01cc5ba58723","created":"2026-02-22T19:51:31.088879+0000","modified":"2026-02-22T19:51:38.438635+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:51:31.690292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"89d5fed7-52dc-4e33-8e1e-4c3ec1b58f68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dff05e0-678e-4469-8c56-b41daeb8a87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eafdcdb0-26b9-4071-80a8-29486388a96e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:38.683 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T19:51:39.445 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T19:51:39.440+0000 7f65f6bc5640 -1 osd.0 0 waiting for initial osdmap 2026-02-22T19:51:39.472 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T19:51:39.468+0000 7f65eea09640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:51:40.447 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T19:51:40.444+0000 7f5e16b1e640 -1 osd.2 0 waiting for initial osdmap 2026-02-22T19:51:40.447 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T19:51:40.444+0000 7f8db3cfd640 -1 osd.1 0 waiting for initial osdmap 2026-02-22T19:51:40.464 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T19:51:40.460+0000 7f8dabb41640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:51:40.464 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T19:51:40.460+0000 7f5e0e962640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:51:44.985 DEBUG:teuthology.orchestra.run.trial042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:45.128 INFO:teuthology.misc.health.trial042.stdout: 2026-02-22T19:51:45.128 INFO:teuthology.misc.health.trial042.stdout:{"epoch":12,"fsid":"de51f049-31b3-4825-9b8c-01cc5ba58723","created":"2026-02-22T19:51:31.088879+0000","modified":"2026-02-22T19:51:44.452897+0000","last_up_change":"2026-02-22T19:51:41.445931+0000","last_in_change":"2026-02-22T19:51:31.690292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:51:43.260639+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"89d5fed7-52dc-4e33-8e1e-4c3ec1b58f68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6802","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6803","nonce":1163499899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6804","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6805","nonce":1163499899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6808","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6809","nonce":1163499899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6806","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6807","nonce":1163499899}]},"public_addr":"10.20.193.42:6803/1163499899","cluster_addr":"10.20.193.42:6805/1163499899","heartbeat_back_addr":"10.20.193.42:6809/1163499899","heartbeat_front_addr":"10.20.193.42:6807/1163499899","state":["exists","up"]},{"osd":1,"uuid":"3dff05e0-678e-4469-8c56-b41daeb8a87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6810","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6811","nonce":1923107216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6812","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6813","nonce":1923107216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6816","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6817","nonce":1923107216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6814","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6815","nonce":1923107216}]},"public_addr":"10.20.193.42:6811/1923107216","cluster_addr":"10.20.193.42:6813/1923107216","heartbeat_back_addr":"10.20.193.42:6817/1923107216","heartbeat_front_addr":"10.20.193.42:6815/1923107216","state":["exists","up"]},{"osd":2,"uuid":"eafdcdb0-26b9-4071-80a8-29486388a96e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6818","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6819","nonce":1307521726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6820","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6821","nonce":1307521726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6824","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6825","nonce":1307521726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6822","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6823","nonce":1307521726}]},"public_addr":"10.20.193.42:6819/1307521726","cluster_addr":"10.20.193.42:6821/1307521726","heartbeat_back_addr":"10.20.193.42:6825/1307521726","heartbeat_front_addr":"10.20.193.42:6823/1307521726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.357125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.513975+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.630821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:45.137 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-22T19:51:45.138 INFO:tasks.ceph:Creating RBD pool 2026-02-22T19:51:45.138 DEBUG:teuthology.orchestra.run.trial042:> sudo ceph --cluster cluster1 osd pool create rbd 8 2026-02-22T19:51:45.456 INFO:teuthology.orchestra.run.trial042.stderr:pool 'rbd' created 2026-02-22T19:51:45.470 DEBUG:teuthology.orchestra.run.trial042:> rbd --cluster cluster1 pool init rbd 2026-02-22T19:51:48.471 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2026-02-22T19:51:48.472 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2026-02-22T19:51:48.472 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2026-02-22T19:51:48.472 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2026-02-22T19:51:48.473 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-22T19:51:48.643 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:48.652 INFO:teuthology.orchestra.run.trial042.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6800","nonce":4114566765},{"type":"v1","addr":"10.20.193.42:6801","nonce":4114566765}]},"active_addr":"10.20.193.42:6801/4114566765","active_change":"2026-02-22T19:51:33.220585+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.42:0","nonce":42643023}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.42:0","nonce":973699167}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.42:0","nonce":1048379912}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.42:0","nonce":3806594531}]}]} 2026-02-22T19:51:48.659 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2026-02-22T19:51:48.660 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2026-02-22T19:51:48.660 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:48.810 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:48.810 INFO:teuthology.orchestra.run.trial042.stdout:{"epoch":16,"fsid":"de51f049-31b3-4825-9b8c-01cc5ba58723","created":"2026-02-22T19:51:31.088879+0000","modified":"2026-02-22T19:51:48.461714+0000","last_up_change":"2026-02-22T19:51:41.445931+0000","last_in_change":"2026-02-22T19:51:31.690292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:51:43.260639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:51:45.285569+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"89d5fed7-52dc-4e33-8e1e-4c3ec1b58f68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6802","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6803","nonce":1163499899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6804","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6805","nonce":1163499899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6808","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6809","nonce":1163499899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6806","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6807","nonce":1163499899}]},"public_addr":"10.20.193.42:6803/1163499899","cluster_addr":"10.20.193.42:6805/1163499899","heartbeat_back_addr":"10.20.193.42:6809/1163499899","heartbeat_front_addr":"10.20.193.42:6807/1163499899","state":["exists","up"]},{"osd":1,"uuid":"3dff05e0-678e-4469-8c56-b41daeb8a87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6810","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6811","nonce":1923107216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6812","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6813","nonce":1923107216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6816","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6817","nonce":1923107216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6814","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6815","nonce":1923107216}]},"public_addr":"10.20.193.42:6811/1923107216","cluster_addr":"10.20.193.42:6813/1923107216","heartbeat_back_addr":"10.20.193.42:6817/1923107216","heartbeat_front_addr":"10.20.193.42:6815/1923107216","state":["exists","up"]},{"osd":2,"uuid":"eafdcdb0-26b9-4071-80a8-29486388a96e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6818","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6819","nonce":1307521726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6820","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6821","nonce":1307521726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6824","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6825","nonce":1307521726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6822","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6823","nonce":1307521726}]},"public_addr":"10.20.193.42:6819/1307521726","cluster_addr":"10.20.193.42:6821/1307521726","heartbeat_back_addr":"10.20.193.42:6825/1307521726","heartbeat_front_addr":"10.20.193.42:6823/1307521726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.357125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.513975+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.630821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:48.821 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2026-02-22T19:51:48.821 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:48.966 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:48.967 INFO:teuthology.orchestra.run.trial042.stdout:{"epoch":16,"fsid":"de51f049-31b3-4825-9b8c-01cc5ba58723","created":"2026-02-22T19:51:31.088879+0000","modified":"2026-02-22T19:51:48.461714+0000","last_up_change":"2026-02-22T19:51:41.445931+0000","last_in_change":"2026-02-22T19:51:31.690292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:51:43.260639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:51:45.285569+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"89d5fed7-52dc-4e33-8e1e-4c3ec1b58f68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6802","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6803","nonce":1163499899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6804","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6805","nonce":1163499899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6808","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6809","nonce":1163499899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6806","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6807","nonce":1163499899}]},"public_addr":"10.20.193.42:6803/1163499899","cluster_addr":"10.20.193.42:6805/1163499899","heartbeat_back_addr":"10.20.193.42:6809/1163499899","heartbeat_front_addr":"10.20.193.42:6807/1163499899","state":["exists","up"]},{"osd":1,"uuid":"3dff05e0-678e-4469-8c56-b41daeb8a87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6810","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6811","nonce":1923107216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6812","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6813","nonce":1923107216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6816","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6817","nonce":1923107216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6814","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6815","nonce":1923107216}]},"public_addr":"10.20.193.42:6811/1923107216","cluster_addr":"10.20.193.42:6813/1923107216","heartbeat_back_addr":"10.20.193.42:6817/1923107216","heartbeat_front_addr":"10.20.193.42:6815/1923107216","state":["exists","up"]},{"osd":2,"uuid":"eafdcdb0-26b9-4071-80a8-29486388a96e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6818","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6819","nonce":1307521726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6820","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6821","nonce":1307521726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6824","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6825","nonce":1307521726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6822","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6823","nonce":1307521726}]},"public_addr":"10.20.193.42:6819/1307521726","cluster_addr":"10.20.193.42:6821/1307521726","heartbeat_back_addr":"10.20.193.42:6825/1307521726","heartbeat_front_addr":"10.20.193.42:6823/1307521726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.357125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.513975+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.630821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:48.980 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2026-02-22T19:51:48.980 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2026-02-22T19:51:48.981 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2026-02-22T19:51:49.065 INFO:teuthology.orchestra.run.trial042.stdout:34359738371 2026-02-22T19:51:49.066 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2026-02-22T19:51:49.067 INFO:teuthology.orchestra.run.trial042.stdout:38654705667 2026-02-22T19:51:49.068 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2026-02-22T19:51:49.069 INFO:teuthology.orchestra.run.trial042.stdout:38654705667 2026-02-22T19:51:49.069 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2026-02-22T19:51:49.254 INFO:teuthology.orchestra.run.trial042.stdout:38654705665 2026-02-22T19:51:49.258 INFO:teuthology.orchestra.run.trial042.stdout:34359738370 2026-02-22T19:51:49.259 INFO:teuthology.orchestra.run.trial042.stdout:38654705665 2026-02-22T19:51:49.265 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705665 for osd.1 2026-02-22T19:51:49.270 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2026-02-22T19:51:49.272 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705665 for osd.2 2026-02-22T19:51:50.266 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2026-02-22T19:51:50.271 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2026-02-22T19:51:50.272 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2026-02-22T19:51:50.421 INFO:teuthology.orchestra.run.trial042.stdout:38654705667 2026-02-22T19:51:50.434 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705667 for osd.1 2026-02-22T19:51:50.434 DEBUG:teuthology.parallel:result is None 2026-02-22T19:51:50.454 INFO:teuthology.orchestra.run.trial042.stdout:38654705667 2026-02-22T19:51:50.455 INFO:teuthology.orchestra.run.trial042.stdout:34359738371 2026-02-22T19:51:50.465 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705667 for osd.2 2026-02-22T19:51:50.465 DEBUG:teuthology.parallel:result is None 2026-02-22T19:51:50.467 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2026-02-22T19:51:50.468 DEBUG:teuthology.parallel:result is None 2026-02-22T19:51:50.468 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2026-02-22T19:51:50.468 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T19:51:50.654 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:50.655 INFO:teuthology.orchestra.run.trial042.stderr:dumped all 2026-02-22T19:51:50.664 INFO:teuthology.orchestra.run.trial042.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-22T19:51:49.224497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82956,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283032564,"statfs":{"total":289910292480,"available":289825345536,"internally_reserved":0,"allocated":1589248,"data_stored":1291059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.762689"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467456+0000","last_change":"2026-02-22T19:51:48.467655+0000","last_active":"2026-02-22T19:51:48.467456+0000","last_peered":"2026-02-22T19:51:48.467456+0000","last_clean":"2026-02-22T19:51:48.467456+0000","last_became_active":"2026-02-22T19:51:46.461368+0000","last_became_peered":"2026-02-22T19:51:46.461368+0000","last_unstale":"2026-02-22T19:51:48.467456+0000","last_undegraded":"2026-02-22T19:51:48.467456+0000","last_fullsized":"2026-02-22T19:51:48.467456+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:54:49.652425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059369800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467246+0000","last_change":"2026-02-22T19:51:48.467546+0000","last_active":"2026-02-22T19:51:48.467246+0000","last_peered":"2026-02-22T19:51:48.467246+0000","last_clean":"2026-02-22T19:51:48.467246+0000","last_became_active":"2026-02-22T19:51:46.461805+0000","last_became_peered":"2026-02-22T19:51:46.461805+0000","last_unstale":"2026-02-22T19:51:48.467246+0000","last_undegraded":"2026-02-22T19:51:48.467246+0000","last_fullsized":"2026-02-22T19:51:48.467246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:40:08.170741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048713899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467120+0000","last_change":"2026-02-22T19:51:48.467382+0000","last_active":"2026-02-22T19:51:48.467120+0000","last_peered":"2026-02-22T19:51:48.467120+0000","last_clean":"2026-02-22T19:51:48.467120+0000","last_became_active":"2026-02-22T19:51:46.461823+0000","last_became_peered":"2026-02-22T19:51:46.461823+0000","last_unstale":"2026-02-22T19:51:48.467120+0000","last_undegraded":"2026-02-22T19:51:48.467120+0000","last_fullsized":"2026-02-22T19:51:48.467120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:03:33.088966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000469977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467838+0000","last_change":"2026-02-22T19:51:48.468007+0000","last_active":"2026-02-22T19:51:48.467838+0000","last_peered":"2026-02-22T19:51:48.467838+0000","last_clean":"2026-02-22T19:51:48.467838+0000","last_became_active":"2026-02-22T19:51:46.461360+0000","last_became_peered":"2026-02-22T19:51:46.461360+0000","last_unstale":"2026-02-22T19:51:48.467838+0000","last_undegraded":"2026-02-22T19:51:48.467838+0000","last_fullsized":"2026-02-22T19:51:48.467838+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:35:55.755801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000927201,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.465651+0000","last_change":"2026-02-22T19:51:48.465651+0000","last_active":"2026-02-22T19:51:48.465651+0000","last_peered":"2026-02-22T19:51:48.465651+0000","last_clean":"2026-02-22T19:51:48.465651+0000","last_became_active":"2026-02-22T19:51:46.461880+0000","last_became_peered":"2026-02-22T19:51:46.461880+0000","last_unstale":"2026-02-22T19:51:48.465651+0000","last_undegraded":"2026-02-22T19:51:48.465651+0000","last_fullsized":"2026-02-22T19:51:48.465651+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:20:16.291609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000706108,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.562584+0000","last_change":"2026-02-22T19:51:48.562746+0000","last_active":"2026-02-22T19:51:48.562584+0000","last_peered":"2026-02-22T19:51:48.562584+0000","last_clean":"2026-02-22T19:51:48.562584+0000","last_became_active":"2026-02-22T19:51:46.461871+0000","last_became_peered":"2026-02-22T19:51:46.461871+0000","last_unstale":"2026-02-22T19:51:48.562584+0000","last_undegraded":"2026-02-22T19:51:48.562584+0000","last_fullsized":"2026-02-22T19:51:48.562584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:23.013120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028334900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.562558+0000","last_change":"2026-02-22T19:51:48.562686+0000","last_active":"2026-02-22T19:51:48.562558+0000","last_peered":"2026-02-22T19:51:48.562558+0000","last_clean":"2026-02-22T19:51:48.562558+0000","last_became_active":"2026-02-22T19:51:46.461861+0000","last_became_peered":"2026-02-22T19:51:46.461861+0000","last_unstale":"2026-02-22T19:51:48.562558+0000","last_undegraded":"2026-02-22T19:51:48.562558+0000","last_fullsized":"2026-02-22T19:51:48.562558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:28:58.573469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026117799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467598+0000","last_change":"2026-02-22T19:51:48.467890+0000","last_active":"2026-02-22T19:51:48.467598+0000","last_peered":"2026-02-22T19:51:48.467598+0000","last_clean":"2026-02-22T19:51:48.467598+0000","last_became_active":"2026-02-22T19:51:46.461352+0000","last_became_peered":"2026-02-22T19:51:46.461352+0000","last_unstale":"2026-02-22T19:51:48.467598+0000","last_undegraded":"2026-02-22T19:51:48.467598+0000","last_fullsized":"2026-02-22T19:51:48.467598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:01:13.797556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082900699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'72","reported_seq":108,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.466741+0000","last_change":"2026-02-22T19:51:45.322266+0000","last_active":"2026-02-22T19:51:48.466741+0000","last_peered":"2026-02-22T19:51:48.466741+0000","last_clean":"2026-02-22T19:51:48.466741+0000","last_became_active":"2026-02-22T19:51:45.321492+0000","last_became_peered":"2026-02-22T19:51:45.321492+0000","last_unstale":"2026-02-22T19:51:48.466741+0000","last_undegraded":"2026-02-22T19:51:48.466741+0000","last_fullsized":"2026-02-22T19:51:48.466741+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_clean_scrub_stamp":"2026-02-22T19:51:43.450496+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:22:35.569483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":131072,"data_stored":36769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":729088,"data_stored":627145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":729088,"data_stored":627145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T19:51:50.666 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T19:51:50.818 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:50.819 INFO:teuthology.orchestra.run.trial042.stderr:dumped all 2026-02-22T19:51:50.830 INFO:teuthology.orchestra.run.trial042.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-22T19:51:49.224497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82956,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283032564,"statfs":{"total":289910292480,"available":289825345536,"internally_reserved":0,"allocated":1589248,"data_stored":1291059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.762689"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467456+0000","last_change":"2026-02-22T19:51:48.467655+0000","last_active":"2026-02-22T19:51:48.467456+0000","last_peered":"2026-02-22T19:51:48.467456+0000","last_clean":"2026-02-22T19:51:48.467456+0000","last_became_active":"2026-02-22T19:51:46.461368+0000","last_became_peered":"2026-02-22T19:51:46.461368+0000","last_unstale":"2026-02-22T19:51:48.467456+0000","last_undegraded":"2026-02-22T19:51:48.467456+0000","last_fullsized":"2026-02-22T19:51:48.467456+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:54:49.652425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059369800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467246+0000","last_change":"2026-02-22T19:51:48.467546+0000","last_active":"2026-02-22T19:51:48.467246+0000","last_peered":"2026-02-22T19:51:48.467246+0000","last_clean":"2026-02-22T19:51:48.467246+0000","last_became_active":"2026-02-22T19:51:46.461805+0000","last_became_peered":"2026-02-22T19:51:46.461805+0000","last_unstale":"2026-02-22T19:51:48.467246+0000","last_undegraded":"2026-02-22T19:51:48.467246+0000","last_fullsized":"2026-02-22T19:51:48.467246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:40:08.170741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048713899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467120+0000","last_change":"2026-02-22T19:51:48.467382+0000","last_active":"2026-02-22T19:51:48.467120+0000","last_peered":"2026-02-22T19:51:48.467120+0000","last_clean":"2026-02-22T19:51:48.467120+0000","last_became_active":"2026-02-22T19:51:46.461823+0000","last_became_peered":"2026-02-22T19:51:46.461823+0000","last_unstale":"2026-02-22T19:51:48.467120+0000","last_undegraded":"2026-02-22T19:51:48.467120+0000","last_fullsized":"2026-02-22T19:51:48.467120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:03:33.088966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000469977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467838+0000","last_change":"2026-02-22T19:51:48.468007+0000","last_active":"2026-02-22T19:51:48.467838+0000","last_peered":"2026-02-22T19:51:48.467838+0000","last_clean":"2026-02-22T19:51:48.467838+0000","last_became_active":"2026-02-22T19:51:46.461360+0000","last_became_peered":"2026-02-22T19:51:46.461360+0000","last_unstale":"2026-02-22T19:51:48.467838+0000","last_undegraded":"2026-02-22T19:51:48.467838+0000","last_fullsized":"2026-02-22T19:51:48.467838+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:35:55.755801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000927201,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.465651+0000","last_change":"2026-02-22T19:51:48.465651+0000","last_active":"2026-02-22T19:51:48.465651+0000","last_peered":"2026-02-22T19:51:48.465651+0000","last_clean":"2026-02-22T19:51:48.465651+0000","last_became_active":"2026-02-22T19:51:46.461880+0000","last_became_peered":"2026-02-22T19:51:46.461880+0000","last_unstale":"2026-02-22T19:51:48.465651+0000","last_undegraded":"2026-02-22T19:51:48.465651+0000","last_fullsized":"2026-02-22T19:51:48.465651+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:20:16.291609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000706108,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.562584+0000","last_change":"2026-02-22T19:51:48.562746+0000","last_active":"2026-02-22T19:51:48.562584+0000","last_peered":"2026-02-22T19:51:48.562584+0000","last_clean":"2026-02-22T19:51:48.562584+0000","last_became_active":"2026-02-22T19:51:46.461871+0000","last_became_peered":"2026-02-22T19:51:46.461871+0000","last_unstale":"2026-02-22T19:51:48.562584+0000","last_undegraded":"2026-02-22T19:51:48.562584+0000","last_fullsized":"2026-02-22T19:51:48.562584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:23.013120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028334900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.562558+0000","last_change":"2026-02-22T19:51:48.562686+0000","last_active":"2026-02-22T19:51:48.562558+0000","last_peered":"2026-02-22T19:51:48.562558+0000","last_clean":"2026-02-22T19:51:48.562558+0000","last_became_active":"2026-02-22T19:51:46.461861+0000","last_became_peered":"2026-02-22T19:51:46.461861+0000","last_unstale":"2026-02-22T19:51:48.562558+0000","last_undegraded":"2026-02-22T19:51:48.562558+0000","last_fullsized":"2026-02-22T19:51:48.562558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:28:58.573469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026117799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.467598+0000","last_change":"2026-02-22T19:51:48.467890+0000","last_active":"2026-02-22T19:51:48.467598+0000","last_peered":"2026-02-22T19:51:48.467598+0000","last_clean":"2026-02-22T19:51:48.467598+0000","last_became_active":"2026-02-22T19:51:46.461352+0000","last_became_peered":"2026-02-22T19:51:46.461352+0000","last_unstale":"2026-02-22T19:51:48.467598+0000","last_undegraded":"2026-02-22T19:51:48.467598+0000","last_fullsized":"2026-02-22T19:51:48.467598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:01:13.797556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082900699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'72","reported_seq":108,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.466741+0000","last_change":"2026-02-22T19:51:45.322266+0000","last_active":"2026-02-22T19:51:48.466741+0000","last_peered":"2026-02-22T19:51:48.466741+0000","last_clean":"2026-02-22T19:51:48.466741+0000","last_became_active":"2026-02-22T19:51:45.321492+0000","last_became_peered":"2026-02-22T19:51:45.321492+0000","last_unstale":"2026-02-22T19:51:48.466741+0000","last_undegraded":"2026-02-22T19:51:48.466741+0000","last_fullsized":"2026-02-22T19:51:48.466741+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_clean_scrub_stamp":"2026-02-22T19:51:43.450496+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:22:35.569483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":131072,"data_stored":36769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":729088,"data_stored":627145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":729088,"data_stored":627145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T19:51:50.831 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2026-02-22T19:51:50.832 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2026-02-22T19:51:50.832 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2026-02-22T19:51:50.832 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2026-02-22T19:51:50.997 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:50.997 INFO:teuthology.orchestra.run.trial042.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-22T19:51:51.007 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2026-02-22T19:51:51.008 INFO:teuthology.run_tasks:Running task ceph... 2026-02-22T19:51:51.021 INFO:tasks.ceph:Creating ceph cluster cluster2... 2026-02-22T19:51:51.022 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-22T19:51:51.022 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972', 'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '63972', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9196, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'targets': {'trial042.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJFOY7gCzruyFRuLBxtBoDvQgtaYFFQOw5vx3RBuU1mB/Ah/xwnZFQzE4e9LqJjI8XDxcQI6v5hZ7Xf3q0Mv6lg=', 'trial066.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdYkUMhYRZ+Fncb1nPoj2MOd3bMQdmVaMHKfRqH7QQahTfDkUNFuUFQ6ruWGi0llCxilrcV5oFjpJ06/7W/PBc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'pool', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'journal', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-22_19:35:38', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-22T19:51:51.023 DEBUG:teuthology.orchestra.run.trial042:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2026-02-22T19:51:51.025 DEBUG:teuthology.orchestra.run.trial066:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2026-02-22T19:51:51.031 DEBUG:teuthology.orchestra.run.trial042:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T19:51:51.072 DEBUG:teuthology.orchestra.run.trial066:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T19:51:51.081 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:51.081 DEBUG:teuthology.orchestra.run.trial066:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T19:51:51.126 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T19:51:51.126 DEBUG:teuthology.orchestra.run.trial066:> stat /dev/vg_nvme/lv_1 2026-02-22T19:51:51.173 INFO:teuthology.orchestra.run.trial066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T19:51:51.174 INFO:teuthology.orchestra.run.trial066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:51.174 INFO:teuthology.orchestra.run.trial066.stdout:Device: 5h/5d Inode: 905 Links: 1 2026-02-22T19:51:51.174 INFO:teuthology.orchestra.run.trial066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:51.174 INFO:teuthology.orchestra.run.trial066.stdout:Access: 2026-02-22 19:46:43.639629065 +0000 2026-02-22T19:51:51.174 INFO:teuthology.orchestra.run.trial066.stdout:Modify: 2026-02-22 19:46:43.451632262 +0000 2026-02-22T19:51:51.174 INFO:teuthology.orchestra.run.trial066.stdout:Change: 2026-02-22 19:46:43.451632262 +0000 2026-02-22T19:51:51.174 INFO:teuthology.orchestra.run.trial066.stdout: Birth: - 2026-02-22T19:51:51.174 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T19:51:51.225 INFO:teuthology.orchestra.run.trial066.stderr:1+0 records in 2026-02-22T19:51:51.225 INFO:teuthology.orchestra.run.trial066.stderr:1+0 records out 2026-02-22T19:51:51.225 INFO:teuthology.orchestra.run.trial066.stderr:512 bytes copied, 0.000188053 s, 2.7 MB/s 2026-02-22T19:51:51.226 DEBUG:teuthology.orchestra.run.trial066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T19:51:51.275 DEBUG:teuthology.orchestra.run.trial066:> stat /dev/vg_nvme/lv_2 2026-02-22T19:51:51.321 INFO:teuthology.orchestra.run.trial066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T19:51:51.321 INFO:teuthology.orchestra.run.trial066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:51.321 INFO:teuthology.orchestra.run.trial066.stdout:Device: 5h/5d Inode: 928 Links: 1 2026-02-22T19:51:51.321 INFO:teuthology.orchestra.run.trial066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:51.322 INFO:teuthology.orchestra.run.trial066.stdout:Access: 2026-02-22 19:46:43.979623282 +0000 2026-02-22T19:51:51.322 INFO:teuthology.orchestra.run.trial066.stdout:Modify: 2026-02-22 19:46:43.811626141 +0000 2026-02-22T19:51:51.322 INFO:teuthology.orchestra.run.trial066.stdout:Change: 2026-02-22 19:46:43.811626141 +0000 2026-02-22T19:51:51.322 INFO:teuthology.orchestra.run.trial066.stdout: Birth: - 2026-02-22T19:51:51.322 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T19:51:51.368 INFO:teuthology.orchestra.run.trial066.stderr:1+0 records in 2026-02-22T19:51:51.368 INFO:teuthology.orchestra.run.trial066.stderr:1+0 records out 2026-02-22T19:51:51.369 INFO:teuthology.orchestra.run.trial066.stderr:512 bytes copied, 0.000162484 s, 3.2 MB/s 2026-02-22T19:51:51.370 DEBUG:teuthology.orchestra.run.trial066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T19:51:51.419 DEBUG:teuthology.orchestra.run.trial066:> stat /dev/vg_nvme/lv_3 2026-02-22T19:51:51.465 INFO:teuthology.orchestra.run.trial066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T19:51:51.465 INFO:teuthology.orchestra.run.trial066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:51.465 INFO:teuthology.orchestra.run.trial066.stdout:Device: 5h/5d Inode: 953 Links: 1 2026-02-22T19:51:51.465 INFO:teuthology.orchestra.run.trial066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:51.466 INFO:teuthology.orchestra.run.trial066.stdout:Access: 2026-02-22 19:46:44.295617911 +0000 2026-02-22T19:51:51.466 INFO:teuthology.orchestra.run.trial066.stdout:Modify: 2026-02-22 19:46:44.127620767 +0000 2026-02-22T19:51:51.466 INFO:teuthology.orchestra.run.trial066.stdout:Change: 2026-02-22 19:46:44.127620767 +0000 2026-02-22T19:51:51.466 INFO:teuthology.orchestra.run.trial066.stdout: Birth: - 2026-02-22T19:51:51.466 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T19:51:51.512 INFO:teuthology.orchestra.run.trial066.stderr:1+0 records in 2026-02-22T19:51:51.513 INFO:teuthology.orchestra.run.trial066.stderr:1+0 records out 2026-02-22T19:51:51.513 INFO:teuthology.orchestra.run.trial066.stderr:512 bytes copied, 0.000172344 s, 3.0 MB/s 2026-02-22T19:51:51.514 DEBUG:teuthology.orchestra.run.trial066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T19:51:51.564 DEBUG:teuthology.orchestra.run.trial066:> stat /dev/vg_nvme/lv_4 2026-02-22T19:51:51.609 INFO:teuthology.orchestra.run.trial066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T19:51:51.609 INFO:teuthology.orchestra.run.trial066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:51.610 INFO:teuthology.orchestra.run.trial066.stdout:Device: 5h/5d Inode: 976 Links: 1 2026-02-22T19:51:51.610 INFO:teuthology.orchestra.run.trial066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:51.610 INFO:teuthology.orchestra.run.trial066.stdout:Access: 2026-02-22 19:46:44.615612468 +0000 2026-02-22T19:51:51.610 INFO:teuthology.orchestra.run.trial066.stdout:Modify: 2026-02-22 19:46:44.443615392 +0000 2026-02-22T19:51:51.610 INFO:teuthology.orchestra.run.trial066.stdout:Change: 2026-02-22 19:46:44.443615392 +0000 2026-02-22T19:51:51.610 INFO:teuthology.orchestra.run.trial066.stdout: Birth: - 2026-02-22T19:51:51.610 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T19:51:51.660 INFO:teuthology.orchestra.run.trial066.stderr:1+0 records in 2026-02-22T19:51:51.661 INFO:teuthology.orchestra.run.trial066.stderr:1+0 records out 2026-02-22T19:51:51.661 INFO:teuthology.orchestra.run.trial066.stderr:512 bytes copied, 0.000163356 s, 3.1 MB/s 2026-02-22T19:51:51.662 DEBUG:teuthology.orchestra.run.trial066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T19:51:51.711 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:51.711 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial066.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T19:51:51.711 INFO:tasks.ceph:Generating config... 2026-02-22T19:51:51.713 INFO:tasks.ceph:[client] debug rbd = 20 2026-02-22T19:51:51.713 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2026-02-22T19:51:51.713 INFO:tasks.ceph:[client] log to stderr = False 2026-02-22T19:51:51.713 INFO:tasks.ceph:[client] rbd default features = 125 2026-02-22T19:51:51.714 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2026-02-22T19:51:51.714 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:51.714 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:51.714 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:51.714 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:51.714 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:51.714 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:51.714 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:51.714 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:51.714 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-22T19:51:51.714 INFO:tasks.ceph:[global] mon warn on pool no app = False 2026-02-22T19:51:51.715 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-22T19:51:51.715 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-22T19:51:51.715 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-22T19:51:51.715 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-22T19:51:51.715 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-22T19:51:51.715 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-22T19:51:51.715 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-22T19:51:51.715 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-22T19:51:51.715 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2026-02-22T19:51:51.715 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-22T19:51:51.715 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-22T19:51:51.715 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2026-02-22T19:51:51.716 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2026-02-22T19:51:51.716 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-22T19:51:51.716 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-22T19:51:51.716 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2026-02-22T19:51:51.716 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:51.716 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-22T19:51:51.716 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-22T19:51:51.716 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-22T19:51:51.716 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-22T19:51:51.716 INFO:tasks.ceph:Setting up cluster2.mon.a... 2026-02-22T19:51:51.717 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2026-02-22T19:51:51.769 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster2.keyring 2026-02-22T19:51:51.772 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2026-02-22T19:51:51.832 DEBUG:teuthology.orchestra.run.trial066:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-22T19:51:51.881 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '10.20.193.66')] 2026-02-22T19:51:51.882 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.66', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2026-02-22T19:51:51.883 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:51.883 DEBUG:teuthology.orchestra.run.trial066:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T19:51:51.926 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.66 --print /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:51.986 INFO:teuthology.orchestra.run.trial066.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:51.986 INFO:teuthology.orchestra.run.trial066.stdout:monmaptool: generated fsid 01607384-c85d-4035-8885-8f357ae44922 2026-02-22T19:51:51.986 INFO:teuthology.orchestra.run.trial066.stdout:setting min_mon_release = quincy 2026-02-22T19:51:51.986 INFO:teuthology.orchestra.run.trial066.stdout:epoch 0 2026-02-22T19:51:51.987 INFO:teuthology.orchestra.run.trial066.stdout:fsid 01607384-c85d-4035-8885-8f357ae44922 2026-02-22T19:51:51.987 INFO:teuthology.orchestra.run.trial066.stdout:last_changed 2026-02-22T19:51:51.986111+0000 2026-02-22T19:51:51.987 INFO:teuthology.orchestra.run.trial066.stdout:created 2026-02-22T19:51:51.986111+0000 2026-02-22T19:51:51.987 INFO:teuthology.orchestra.run.trial066.stdout:min_mon_release 17 (quincy) 2026-02-22T19:51:51.987 INFO:teuthology.orchestra.run.trial066.stdout:election_strategy: 1 2026-02-22T19:51:51.987 INFO:teuthology.orchestra.run.trial066.stdout:0: [v2:10.20.193.66:3300/0,v1:10.20.193.66:6789/0] mon.a 2026-02-22T19:51:51.987 INFO:teuthology.orchestra.run.trial066.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2026-02-22T19:51:51.988 DEBUG:teuthology.orchestra.run.trial066:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T19:51:52.033 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 01607384-c85d-4035-8885-8f357ae44922... 2026-02-22T19:51:52.034 DEBUG:teuthology.orchestra.run.trial042:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2026-02-22T19:51:52.036 DEBUG:teuthology.orchestra.run.trial066:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2026-02-22T19:51:52.076 INFO:teuthology.orchestra.run.trial042.stdout:[global] 2026-02-22T19:51:52.076 INFO:teuthology.orchestra.run.trial042.stdout: chdir = "" 2026-02-22T19:51:52.076 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T19:51:52.076 INFO:teuthology.orchestra.run.trial042.stdout: auth supported = cephx 2026-02-22T19:51:52.076 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.076 INFO:teuthology.orchestra.run.trial042.stdout: filestore xattr use omap = true 2026-02-22T19:51:52.076 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.076 INFO:teuthology.orchestra.run.trial042.stdout: mon clock drift allowed = 1.000 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: osd crush chooseleaf type = 0 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: auth debug = true 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: ms die on old message = true 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: ms die on bug = true 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: mon pg warn max object skew = 0 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.077 INFO:teuthology.orchestra.run.trial042.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: osd pool default size = 2 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: mon osd allow primary affinity = true 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: mon osd allow pg remap = true 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on legacy crush tunables = false 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on crush straw calc version zero = false 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on no sortbitwise = false 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on osd down out interval zero = false 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on too few osds = false 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T19:51:52.078 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: mon_allow_pool_size_one = true 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: osd default data pool replay window = 5 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: mon allow pool delete = true 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: mon cluster log file level = debug 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: debug asserts on shutdown = true 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: mon health detail to clog = false 2026-02-22T19:51:52.079 INFO:teuthology.orchestra.run.trial042.stdout: mon host = 10.20.193.66 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: mon client directed command retry = 5 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: mon warn on pool no app = False 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: ms inject socket failures = 5000 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: fsid = 01607384-c85d-4035-8885-8f357ae44922 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout:[osd] 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: osd journal size = 100 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: osd scrub load threshold = 5.0 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: osd scrub max interval = 600 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: osd mclock profile = high_recovery_ops 2026-02-22T19:51:52.080 INFO:teuthology.orchestra.run.trial042.stdout: osd mclock skip benchmark = true 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: osd recover clone overlap = true 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: osd recovery max chunk = 1048576 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: osd debug shutdown = true 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: osd debug op order = true 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: osd debug verify stray on activate = true 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: osd debug trim objects = true 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: osd open classes on start = true 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: osd debug pg log writeout = true 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.081 INFO:teuthology.orchestra.run.trial042.stdout: osd deep scrub update digest min age = 30 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: osd map max advance = 10 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: journal zero on create = true 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: filestore ondisk finisher threads = 3 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: filestore apply finisher threads = 3 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: bdev debug aio = true 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: osd debug misdirected ops = true 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: bdev async discard = True 2026-02-22T19:51:52.082 INFO:teuthology.orchestra.run.trial042.stdout: bdev enable discard = True 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: bluestore allocator = stupid 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: bluestore block size = 96636764160 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: bluestore fsck on mount = True 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: debug bluefs = 1/20 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: debug bluestore = 1/20 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: debug ms = 1 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: debug osd = 20 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: debug rocksdb = 4/10 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: mon osd full ratio = 0.9 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: mon osd nearfull ratio = 0.8 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: osd failsafe full ratio = 0.95 2026-02-22T19:51:52.083 INFO:teuthology.orchestra.run.trial042.stdout: osd objectstore = bluestore 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout:[mgr] 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: debug ms = 1 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: debug mgr = 20 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: debug mon = 20 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: debug auth = 20 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: mgr/telemetry/nag = false 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout:[mon] 2026-02-22T19:51:52.084 INFO:teuthology.orchestra.run.trial042.stdout: debug ms = 1 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: debug mon = 20 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: debug paxos = 20 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: debug auth = 20 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: mon data avail warn = 5 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: mon mgr mkfs grace = 240 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: mon osd reporter subtree level = osd 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: mon osd prime pg temp = true 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T19:51:52.085 INFO:teuthology.orchestra.run.trial042.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: # 1m isn't quite enough 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: mon_down_mkfs_grace = 2m 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: mon_warn_on_filestore_osds = false 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout:[client] 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: rgw cache enabled = true 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: rgw enable ops log = true 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: rgw enable usage log = true 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: debug rbd = 20 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: debug rbd_mirror = 15 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: log to stderr = False 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: rbd default features = 125 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout: rbd mirror image policy type = simple 2026-02-22T19:51:52.087 INFO:teuthology.orchestra.run.trial042.stdout:[mon.a] 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.0] 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.1] 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.2] 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout:[client.mirror.3] 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.088 INFO:teuthology.orchestra.run.trial042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.092 INFO:teuthology.orchestra.run.trial066.stdout:[global] 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: chdir = "" 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: auth supported = cephx 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: filestore xattr use omap = true 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: mon clock drift allowed = 1.000 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: osd crush chooseleaf type = 0 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: auth debug = true 2026-02-22T19:51:52.093 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: ms die on old message = true 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: ms die on bug = true 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: mon pg warn max object skew = 0 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: osd pool default size = 2 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.094 INFO:teuthology.orchestra.run.trial066.stdout: mon osd allow primary affinity = true 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: mon osd allow pg remap = true 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on legacy crush tunables = false 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on crush straw calc version zero = false 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on no sortbitwise = false 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on osd down out interval zero = false 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on too few osds = false 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: mon_allow_pool_size_one = true 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T19:51:52.095 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: osd default data pool replay window = 5 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: mon allow pool delete = true 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: mon cluster log file level = debug 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: debug asserts on shutdown = true 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: mon health detail to clog = false 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: mon host = 10.20.193.66 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: mon client directed command retry = 5 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: mon warn on pool no app = False 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: ms inject socket failures = 5000 2026-02-22T19:51:52.096 INFO:teuthology.orchestra.run.trial066.stdout: fsid = 01607384-c85d-4035-8885-8f357ae44922 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout:[osd] 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: osd journal size = 100 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: osd scrub load threshold = 5.0 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: osd scrub max interval = 600 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: osd mclock profile = high_recovery_ops 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: osd mclock skip benchmark = true 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: osd recover clone overlap = true 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: osd recovery max chunk = 1048576 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.097 INFO:teuthology.orchestra.run.trial066.stdout: osd debug shutdown = true 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: osd debug op order = true 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: osd debug verify stray on activate = true 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: osd debug trim objects = true 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: osd open classes on start = true 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: osd debug pg log writeout = true 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: osd deep scrub update digest min age = 30 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: osd map max advance = 10 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.098 INFO:teuthology.orchestra.run.trial066.stdout: journal zero on create = true 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: filestore ondisk finisher threads = 3 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: filestore apply finisher threads = 3 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: bdev debug aio = true 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: osd debug misdirected ops = true 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: bdev async discard = True 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: bdev enable discard = True 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: bluestore allocator = stupid 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: bluestore block size = 96636764160 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: bluestore fsck on mount = True 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: debug bluefs = 1/20 2026-02-22T19:51:52.099 INFO:teuthology.orchestra.run.trial066.stdout: debug bluestore = 1/20 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: debug ms = 1 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: debug osd = 20 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: debug rocksdb = 4/10 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: mon osd full ratio = 0.9 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: mon osd nearfull ratio = 0.8 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: osd failsafe full ratio = 0.95 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: osd objectstore = bluestore 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout:[mgr] 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: debug ms = 1 2026-02-22T19:51:52.100 INFO:teuthology.orchestra.run.trial066.stdout: debug mgr = 20 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: debug mon = 20 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: debug auth = 20 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: mgr/telemetry/nag = false 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout:[mon] 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: debug ms = 1 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: debug mon = 20 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: debug paxos = 20 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: debug auth = 20 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: mon data avail warn = 5 2026-02-22T19:51:52.101 INFO:teuthology.orchestra.run.trial066.stdout: mon mgr mkfs grace = 240 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: mon osd reporter subtree level = osd 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: mon osd prime pg temp = true 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T19:51:52.102 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: # 1m isn't quite enough 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: mon_down_mkfs_grace = 2m 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: mon_warn_on_filestore_osds = false 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout:[client] 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: rgw cache enabled = true 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: rgw enable ops log = true 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: rgw enable usage log = true 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T19:51:52.103 INFO:teuthology.orchestra.run.trial066.stdout: debug rbd = 20 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout: debug rbd_mirror = 15 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout: log to stderr = False 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout: rbd default features = 125 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout: rbd mirror image policy type = simple 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout:[mon.a] 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.0] 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.1] 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.2] 2026-02-22T19:51:52.104 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.105 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.105 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.3] 2026-02-22T19:51:52.105 INFO:teuthology.orchestra.run.trial066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.105 INFO:teuthology.orchestra.run.trial066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.106 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2026-02-22T19:51:52.106 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.171 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-22T19:51:52.171 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:52.172 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2026-02-22T19:51:52.218 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:52.218 DEBUG:teuthology.orchestra.run.trial066:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2026-02-22T19:51:52.267 INFO:tasks.ceph:Sending monmap to node ubuntu@trial042.front.sepia.ceph.com 2026-02-22T19:51:52.267 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:52.267 DEBUG:teuthology.orchestra.run.trial042:> sudo dd of=/etc/ceph/cluster2.keyring 2026-02-22T19:51:52.267 DEBUG:teuthology.orchestra.run.trial042:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.283 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:51:52.284 DEBUG:teuthology.orchestra.run.trial042:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:52.331 INFO:tasks.ceph:Sending monmap to node ubuntu@trial066.front.sepia.ceph.com 2026-02-22T19:51:52.331 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:52.331 DEBUG:teuthology.orchestra.run.trial066:> sudo dd of=/etc/ceph/cluster2.keyring 2026-02-22T19:51:52.331 DEBUG:teuthology.orchestra.run.trial066:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.349 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:51:52.349 DEBUG:teuthology.orchestra.run.trial066:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:52.394 INFO:tasks.ceph:Setting up mon nodes... 2026-02-22T19:51:52.394 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-22T19:51:52.395 DEBUG:teuthology.orchestra.run.trial042:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial042.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2026-02-22T19:51:52.420 INFO:tasks.ceph:Setting up mds nodes... 2026-02-22T19:51:52.420 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-22T19:51:52.420 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2026-02-22T19:51:52.485 INFO:teuthology.orchestra.run.trial042.stdout:creating /etc/ceph/cluster2.client.0.keyring 2026-02-22T19:51:52.492 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2026-02-22T19:51:52.516 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2026-02-22T19:51:52.525 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2026-02-22T19:51:52.588 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2026-02-22T19:51:52.596 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2026-02-22T19:51:52.659 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2026-02-22T19:51:52.669 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2026-02-22T19:51:52.731 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2026-02-22T19:51:52.741 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2026-02-22T19:51:52.803 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2026-02-22T19:51:52.810 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2026-02-22T19:51:52.875 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2026-02-22T19:51:52.883 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2026-02-22T19:51:52.947 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2026-02-22T19:51:52.956 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2026-02-22T19:51:53.019 INFO:teuthology.orchestra.run.trial066.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2026-02-22T19:51:53.027 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-22T19:51:53.027 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial042.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@trial066.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T19:51:53.028 DEBUG:teuthology.orchestra.run.trial066:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2026-02-22T19:51:53.077 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:53.078 INFO:tasks.ceph:role: cluster2.osd.0 2026-02-22T19:51:53.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial066.front.sepia.ceph.com 2026-02-22T19:51:53.078 DEBUG:teuthology.orchestra.run.trial066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T19:51:53.131 INFO:teuthology.orchestra.run.trial066.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:53.131 INFO:teuthology.orchestra.run.trial066.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:53.131 INFO:teuthology.orchestra.run.trial066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:53.131 INFO:teuthology.orchestra.run.trial066.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:53.131 INFO:teuthology.orchestra.run.trial066.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:53.131 INFO:teuthology.orchestra.run.trial066.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:53.132 INFO:teuthology.orchestra.run.trial066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:53.132 INFO:teuthology.orchestra.run.trial066.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:53.132 INFO:teuthology.orchestra.run.trial066.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:53.132 INFO:teuthology.orchestra.run.trial066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:53.707 INFO:teuthology.orchestra.run.trial066.stdout:Discarding blocks...Done. 2026-02-22T19:51:53.709 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial066.front.sepia.ceph.com -o noatime 2026-02-22T19:51:53.709 DEBUG:teuthology.orchestra.run.trial066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2026-02-22T19:51:53.766 DEBUG:teuthology.orchestra.run.trial066:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2026-02-22T19:51:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:53.820 INFO:teuthology.orchestra.run.trial066.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:53.821 DEBUG:teuthology.orchestra.run.trial066:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2026-02-22T19:51:53.869 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:53.870 INFO:tasks.ceph:role: cluster2.osd.1 2026-02-22T19:51:53.870 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial066.front.sepia.ceph.com 2026-02-22T19:51:53.870 DEBUG:teuthology.orchestra.run.trial066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T19:51:53.923 INFO:teuthology.orchestra.run.trial066.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:53.923 INFO:teuthology.orchestra.run.trial066.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:53.923 INFO:teuthology.orchestra.run.trial066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:53.923 INFO:teuthology.orchestra.run.trial066.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:53.923 INFO:teuthology.orchestra.run.trial066.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:53.923 INFO:teuthology.orchestra.run.trial066.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:53.923 INFO:teuthology.orchestra.run.trial066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:53.923 INFO:teuthology.orchestra.run.trial066.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:53.924 INFO:teuthology.orchestra.run.trial066.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:53.924 INFO:teuthology.orchestra.run.trial066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:54.489 INFO:teuthology.orchestra.run.trial066.stdout:Discarding blocks...Done. 2026-02-22T19:51:54.491 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial066.front.sepia.ceph.com -o noatime 2026-02-22T19:51:54.491 DEBUG:teuthology.orchestra.run.trial066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2026-02-22T19:51:54.547 DEBUG:teuthology.orchestra.run.trial066:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2026-02-22T19:51:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:54.596 INFO:teuthology.orchestra.run.trial066.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:54.596 DEBUG:teuthology.orchestra.run.trial066:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2026-02-22T19:51:54.646 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:54.646 INFO:tasks.ceph:role: cluster2.osd.2 2026-02-22T19:51:54.646 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial066.front.sepia.ceph.com 2026-02-22T19:51:54.646 DEBUG:teuthology.orchestra.run.trial066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T19:51:54.699 INFO:teuthology.orchestra.run.trial066.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:54.699 INFO:teuthology.orchestra.run.trial066.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:54.700 INFO:teuthology.orchestra.run.trial066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:54.700 INFO:teuthology.orchestra.run.trial066.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:54.700 INFO:teuthology.orchestra.run.trial066.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:54.700 INFO:teuthology.orchestra.run.trial066.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:54.700 INFO:teuthology.orchestra.run.trial066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:54.700 INFO:teuthology.orchestra.run.trial066.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:54.700 INFO:teuthology.orchestra.run.trial066.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:54.700 INFO:teuthology.orchestra.run.trial066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:55.262 INFO:teuthology.orchestra.run.trial066.stdout:Discarding blocks...Done. 2026-02-22T19:51:55.263 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial066.front.sepia.ceph.com -o noatime 2026-02-22T19:51:55.264 DEBUG:teuthology.orchestra.run.trial066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2026-02-22T19:51:55.318 DEBUG:teuthology.orchestra.run.trial066:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2026-02-22T19:51:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:55.367 INFO:teuthology.orchestra.run.trial066.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:55.368 DEBUG:teuthology.orchestra.run.trial066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:55.430 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:51:55.429+0000 7f1e2023d8c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2026-02-22T19:51:55.430 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:51:55.429+0000 7f1e2023d8c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2026-02-22T19:51:55.431 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:51:55.429+0000 7f1e2023d8c0 -1 bdev(0x564661ba5800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:55.431 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:51:55.429+0000 7f1e2023d8c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2026-02-22T19:51:59.497 DEBUG:teuthology.orchestra.run.trial066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2026-02-22T19:51:59.551 DEBUG:teuthology.orchestra.run.trial066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:59.618 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:51:59.617+0000 7fa856b578c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2026-02-22T19:51:59.619 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:51:59.617+0000 7fa856b578c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2026-02-22T19:51:59.619 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:51:59.617+0000 7fa856b578c0 -1 bdev(0x556aae6a1800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:59.619 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:51:59.617+0000 7fa856b578c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2026-02-22T19:52:03.697 DEBUG:teuthology.orchestra.run.trial066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2026-02-22T19:52:03.747 DEBUG:teuthology.orchestra.run.trial066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:52:03.814 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:52:03.813+0000 7fca4b4118c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2026-02-22T19:52:03.815 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:52:03.813+0000 7fca4b4118c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2026-02-22T19:52:03.815 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:52:03.813+0000 7fca4b4118c0 -1 bdev(0x55c5cbd6d800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2026-02-22T19:52:03.815 INFO:teuthology.orchestra.run.trial066.stderr:2026-02-22T19:52:03.813+0000 7fca4b4118c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2026-02-22T19:52:07.885 DEBUG:teuthology.orchestra.run.trial066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2026-02-22T19:52:07.938 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-22T19:52:07.938 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:52:07.939 DEBUG:teuthology.orchestra.run.trial042:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2026-02-22T19:52:07.949 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:07.949 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2026-02-22T19:52:07.990 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:07.990 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2026-02-22T19:52:08.042 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:08.042 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2026-02-22T19:52:08.094 DEBUG:teuthology.orchestra.run.trial042:> set -ex 2026-02-22T19:52:08.094 DEBUG:teuthology.orchestra.run.trial042:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2026-02-22T19:52:08.100 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:08.101 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2026-02-22T19:52:08.142 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:08.142 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2026-02-22T19:52:08.190 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:08.190 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2026-02-22T19:52:08.238 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:08.238 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2026-02-22T19:52:08.286 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:08.286 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2026-02-22T19:52:08.334 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:08.334 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2026-02-22T19:52:08.382 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:08.382 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2026-02-22T19:52:08.430 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:08.430 DEBUG:teuthology.orchestra.run.trial066:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2026-02-22T19:52:08.477 INFO:tasks.ceph:Adding keys to all mons... 2026-02-22T19:52:08.478 DEBUG:teuthology.orchestra.run.trial066:> sudo tee -a /etc/ceph/cluster2.keyring 2026-02-22T19:52:08.528 INFO:teuthology.orchestra.run.trial066.stdout:[mgr.x] 2026-02-22T19:52:08.528 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpwhv2GBAA/MDFvcxaTkRUnQ35IK9xyg== 2026-02-22T19:52:08.528 INFO:teuthology.orchestra.run.trial066.stdout:[osd.0] 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBbXptp/JCtGRAAE0btJbl5ef04m/S6hRi10w== 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout:[osd.1] 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBfXptpx+TgJBAAe1wbZreGALJ3JtCkPX0czw== 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout:[osd.2] 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBjXptpTd+QMBAA0OA5RokVqs/K11IGuHSHfQ== 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout:[client.0] 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpwxPyHBAAjtLthIW/onq21RarJ6XDhw== 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror] 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpDLnRHhAAx4O1JZA/0bYZMH3c2pZQXA== 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.0] 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpdAIRIxAAFkjFfQqZwp3jWzAnNTjbzA== 2026-02-22T19:52:08.529 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.1] 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpMRxYJxAAiT1/lMpChQKlSgF5WthFVw== 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.2] 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpoUmhKxAA6dRMvmXocaaGS3t4z0nCRA== 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.3] 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpx6XlLxAAYqPm5YyrChSCiW1T/JEvbg== 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.4] 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpyFQwNBAAJLCapXf75qed5AeourRCng== 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.5] 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptp2UWEOBAAaYKvVfrfUgXbf5QuTHbppw== 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.6] 2026-02-22T19:52:08.530 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBZXptplbcyARAANNjQK3cdplGnbrbKFpF+pg== 2026-02-22T19:52:08.532 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T19:52:08.596 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:52:08.659 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:52:08.724 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:52:08.786 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:08.851 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:08.915 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:08.979 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:09.042 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:09.106 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:09.170 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:09.234 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:09.298 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:09.362 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-22T19:52:09.363 DEBUG:teuthology.orchestra.run.trial066:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2026-02-22T19:52:09.413 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2026-02-22T19:52:09.486 DEBUG:teuthology.orchestra.run.trial066:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2026-02-22T19:52:09.537 DEBUG:teuthology.orchestra.run.trial066:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:52:09.580 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2026-02-22T19:52:09.580 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2026-02-22T19:52:09.580 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2026-02-22T19:52:09.623 INFO:tasks.ceph.cluster2.mon.a:Started 2026-02-22T19:52:09.623 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2026-02-22T19:52:09.624 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2026-02-22T19:52:09.624 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2026-02-22T19:52:09.626 INFO:tasks.ceph.cluster2.mgr.x:Started 2026-02-22T19:52:09.626 INFO:tasks.ceph:Setting crush tunables to default 2026-02-22T19:52:09.626 DEBUG:teuthology.orchestra.run.trial066:> sudo ceph --cluster cluster2 osd crush tunables default 2026-02-22T19:52:09.796 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:09.760+0000 7fac77001280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T19:52:09.796 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:09.784+0000 7fac77001280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T19:52:09.796 INFO:teuthology.orchestra.run.trial066.stderr:adjusted tunables profile to default 2026-02-22T19:52:09.796 INFO:tasks.ceph:check_enable_crimson: False 2026-02-22T19:52:09.796 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2026-02-22T19:52:09.796 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:09.796 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2026-02-22T19:52:09.806 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:09.806 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2026-02-22T19:52:09.827 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:09.820+0000 7fac77001280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T19:52:09.859 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:09.859 DEBUG:teuthology.orchestra.run.trial066:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2026-02-22T19:52:09.890 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:09.884+0000 7fac77001280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T19:52:09.915 DEBUG:teuthology.orchestra.run.trial066:> sudo ceph --cluster cluster2 osd new 8ec0acef-b32e-4a62-a217-f9fbf183a52a 0 2026-02-22T19:52:09.955 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:09.952+0000 7fac77001280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T19:52:09.992 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:09.988+0000 7fac77001280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T19:52:10.055 INFO:teuthology.orchestra.run.trial066.stdout:0 2026-02-22T19:52:10.059 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:10.056+0000 7fac77001280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T19:52:10.064 DEBUG:teuthology.orchestra.run.trial066:> sudo ceph --cluster cluster2 osd new f352b79a-db0e-4c15-b625-23a8ab9a50b4 1 2026-02-22T19:52:10.159 INFO:teuthology.orchestra.run.trial066.stdout:1 2026-02-22T19:52:10.169 DEBUG:teuthology.orchestra.run.trial066:> sudo ceph --cluster cluster2 osd new 66d84059-d5b1-41a6-9820-620d0015f5bd 2 2026-02-22T19:52:10.220 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:10.216+0000 7fac77001280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T19:52:10.264 INFO:teuthology.orchestra.run.trial066.stdout:2 2026-02-22T19:52:10.274 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2026-02-22T19:52:10.274 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2026-02-22T19:52:10.276 INFO:tasks.ceph.cluster2.osd.0:Started 2026-02-22T19:52:10.276 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2026-02-22T19:52:10.276 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2026-02-22T19:52:10.278 INFO:tasks.ceph.cluster2.osd.1:Started 2026-02-22T19:52:10.278 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2026-02-22T19:52:10.279 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2026-02-22T19:52:10.281 INFO:tasks.ceph.cluster2.osd.2:Started 2026-02-22T19:52:10.281 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:10.375 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:52:10.376 INFO:teuthology.orchestra.run.trial066.stdout:{"epoch":5,"fsid":"01607384-c85d-4035-8885-8f357ae44922","created":"2026-02-22T19:52:09.703535+0000","modified":"2026-02-22T19:52:10.264244+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:52:10.264244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8ec0acef-b32e-4a62-a217-f9fbf183a52a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f352b79a-db0e-4c15-b625-23a8ab9a50b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"66d84059-d5b1-41a6-9820-620d0015f5bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:10.385 INFO:tasks.ceph.ceph_manager.cluster2:[] 2026-02-22T19:52:10.385 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-22T19:52:10.459 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:10.456+0000 7fac77001280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T19:52:10.547 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:10.544+0000 7fac77001280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T19:52:10.573 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:10.568+0000 7fac77001280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T19:52:10.600 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:10.596+0000 7fac77001280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T19:52:10.631 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:10.628+0000 7fac77001280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T19:52:10.687 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:10.725 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:10.720+0000 7fac77001280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T19:52:10.779 INFO:teuthology.misc.health.trial066.stdout: 2026-02-22T19:52:10.779 INFO:teuthology.misc.health.trial066.stdout:{"epoch":5,"fsid":"01607384-c85d-4035-8885-8f357ae44922","created":"2026-02-22T19:52:09.703535+0000","modified":"2026-02-22T19:52:10.264244+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:52:10.264244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8ec0acef-b32e-4a62-a217-f9fbf183a52a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f352b79a-db0e-4c15-b625-23a8ab9a50b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"66d84059-d5b1-41a6-9820-620d0015f5bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:10.787 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T19:52:10.894 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-22T19:52:11.042 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:11.036+0000 7fac77001280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T19:52:11.073 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:11.068+0000 7fac77001280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T19:52:11.146 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:11.140+0000 7fac77001280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T19:52:11.171 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:11.168+0000 7fac77001280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T19:52:11.196 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:11.192+0000 7fac77001280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T19:52:11.221 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:11.216+0000 7fac77001280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T19:52:11.299 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T19:52:11.299 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T19:52:11.299 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr: from numpy import show_config as show_numpy_config 2026-02-22T19:52:11.305 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:11.300+0000 7fac77001280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T19:52:11.338 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:11.332+0000 7fac77001280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T19:52:11.376 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:11.372+0000 7fac77001280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T19:52:12.315 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T19:52:12.313+0000 7f55918b98c0 -1 Falling back to public interface 2026-02-22T19:52:12.323 INFO:tasks.ceph.cluster2.osd.0.trial066.stderr:2026-02-22T19:52:12.321+0000 7f21df6898c0 -1 Falling back to public interface 2026-02-22T19:52:12.334 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T19:52:12.333+0000 7fdfe016a8c0 -1 Falling back to public interface 2026-02-22T19:52:12.401 INFO:tasks.ceph.cluster2.mgr.x.trial042.stderr:2026-02-22T19:52:12.396+0000 7fab95851640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-22T19:52:17.090 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:17.113 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T19:52:17.109+0000 7f55918b98c0 -1 osd.2 0 log_to_monitors true 2026-02-22T19:52:17.205 INFO:tasks.ceph.cluster2.osd.0.trial066.stderr:2026-02-22T19:52:17.201+0000 7f21df6898c0 -1 osd.0 0 log_to_monitors true 2026-02-22T19:52:17.231 INFO:teuthology.misc.health.trial066.stdout: 2026-02-22T19:52:17.232 INFO:teuthology.misc.health.trial066.stdout:{"epoch":6,"fsid":"01607384-c85d-4035-8885-8f357ae44922","created":"2026-02-22T19:52:09.703535+0000","modified":"2026-02-22T19:52:17.165962+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:52:10.264244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8ec0acef-b32e-4a62-a217-f9fbf183a52a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f352b79a-db0e-4c15-b625-23a8ab9a50b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"66d84059-d5b1-41a6-9820-620d0015f5bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:17.242 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T19:52:17.305 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T19:52:17.301+0000 7fdfe016a8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T19:52:18.172 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T19:52:18.169+0000 7f558d840640 -1 osd.2 0 waiting for initial osdmap 2026-02-22T19:52:18.198 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T19:52:18.197+0000 7f5585684640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:52:19.173 INFO:tasks.ceph.cluster2.osd.0.trial066.stderr:2026-02-22T19:52:19.169+0000 7f21db610640 -1 osd.0 0 waiting for initial osdmap 2026-02-22T19:52:19.174 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T19:52:19.173+0000 7fdfdc0f1640 -1 osd.1 0 waiting for initial osdmap 2026-02-22T19:52:19.186 INFO:tasks.ceph.cluster2.osd.0.trial066.stderr:2026-02-22T19:52:19.185+0000 7f21d3454640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:52:19.202 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T19:52:19.201+0000 7fdfd3f35640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:52:23.544 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:23.683 INFO:teuthology.misc.health.trial066.stdout: 2026-02-22T19:52:23.684 INFO:teuthology.misc.health.trial066.stdout:{"epoch":12,"fsid":"01607384-c85d-4035-8885-8f357ae44922","created":"2026-02-22T19:52:09.703535+0000","modified":"2026-02-22T19:52:23.179096+0000","last_up_change":"2026-02-22T19:52:20.171914+0000","last_in_change":"2026-02-22T19:52:10.264244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:52:21.436241+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ec0acef-b32e-4a62-a217-f9fbf183a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6808","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6809","nonce":3981027600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6810","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6811","nonce":3981027600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6814","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6815","nonce":3981027600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6812","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6813","nonce":3981027600}]},"public_addr":"10.20.193.66:6809/3981027600","cluster_addr":"10.20.193.66:6811/3981027600","heartbeat_back_addr":"10.20.193.66:6815/3981027600","heartbeat_front_addr":"10.20.193.66:6813/3981027600","state":["exists","up"]},{"osd":1,"uuid":"f352b79a-db0e-4c15-b625-23a8ab9a50b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6816","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6817","nonce":747693529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6818","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6819","nonce":747693529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6822","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6823","nonce":747693529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6820","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6821","nonce":747693529}]},"public_addr":"10.20.193.66:6817/747693529","cluster_addr":"10.20.193.66:6819/747693529","heartbeat_back_addr":"10.20.193.66:6823/747693529","heartbeat_front_addr":"10.20.193.66:6821/747693529","state":["exists","up"]},{"osd":2,"uuid":"66d84059-d5b1-41a6-9820-620d0015f5bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6800","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6801","nonce":1448960510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6802","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6803","nonce":1448960510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6806","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6807","nonce":1448960510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6804","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6805","nonce":1448960510}]},"public_addr":"10.20.193.66:6801/1448960510","cluster_addr":"10.20.193.66:6803/1448960510","heartbeat_back_addr":"10.20.193.66:6807/1448960510","heartbeat_front_addr":"10.20.193.66:6805/1448960510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.166499+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.348855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.140264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:23.692 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-22T19:52:23.693 INFO:tasks.ceph:Creating RBD pool 2026-02-22T19:52:23.693 DEBUG:teuthology.orchestra.run.trial066:> sudo ceph --cluster cluster2 osd pool create rbd 8 2026-02-22T19:52:24.184 INFO:teuthology.orchestra.run.trial066.stderr:pool 'rbd' created 2026-02-22T19:52:24.195 DEBUG:teuthology.orchestra.run.trial066:> rbd --cluster cluster2 pool init rbd 2026-02-22T19:52:27.198 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2026-02-22T19:52:27.199 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2026-02-22T19:52:27.199 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2026-02-22T19:52:27.199 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2026-02-22T19:52:27.200 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-22T19:52:27.368 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:52:27.378 INFO:teuthology.orchestra.run.trial066.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6826","nonce":3849690483},{"type":"v1","addr":"10.20.193.42:6827","nonce":3849690483}]},"active_addr":"10.20.193.42:6827/3849690483","active_change":"2026-02-22T19:52:11.396232+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.42:0","nonce":2705662875}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.42:0","nonce":3491981135}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.42:0","nonce":1651915373}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.42:0","nonce":2989354799}]}]} 2026-02-22T19:52:27.380 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2026-02-22T19:52:27.380 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2026-02-22T19:52:27.380 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:27.533 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:52:27.533 INFO:teuthology.orchestra.run.trial066.stdout:{"epoch":16,"fsid":"01607384-c85d-4035-8885-8f357ae44922","created":"2026-02-22T19:52:09.703535+0000","modified":"2026-02-22T19:52:27.188585+0000","last_up_change":"2026-02-22T19:52:20.171914+0000","last_in_change":"2026-02-22T19:52:10.264244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:52:21.436241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:52:23.837160+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ec0acef-b32e-4a62-a217-f9fbf183a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6808","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6809","nonce":3981027600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6810","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6811","nonce":3981027600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6814","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6815","nonce":3981027600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6812","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6813","nonce":3981027600}]},"public_addr":"10.20.193.66:6809/3981027600","cluster_addr":"10.20.193.66:6811/3981027600","heartbeat_back_addr":"10.20.193.66:6815/3981027600","heartbeat_front_addr":"10.20.193.66:6813/3981027600","state":["exists","up"]},{"osd":1,"uuid":"f352b79a-db0e-4c15-b625-23a8ab9a50b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6816","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6817","nonce":747693529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6818","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6819","nonce":747693529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6822","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6823","nonce":747693529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6820","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6821","nonce":747693529}]},"public_addr":"10.20.193.66:6817/747693529","cluster_addr":"10.20.193.66:6819/747693529","heartbeat_back_addr":"10.20.193.66:6823/747693529","heartbeat_front_addr":"10.20.193.66:6821/747693529","state":["exists","up"]},{"osd":2,"uuid":"66d84059-d5b1-41a6-9820-620d0015f5bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6800","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6801","nonce":1448960510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6802","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6803","nonce":1448960510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6806","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6807","nonce":1448960510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6804","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6805","nonce":1448960510}]},"public_addr":"10.20.193.66:6801/1448960510","cluster_addr":"10.20.193.66:6803/1448960510","heartbeat_back_addr":"10.20.193.66:6807/1448960510","heartbeat_front_addr":"10.20.193.66:6805/1448960510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.166499+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.348855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.140264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:27.544 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2026-02-22T19:52:27.544 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:27.696 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:52:27.697 INFO:teuthology.orchestra.run.trial066.stdout:{"epoch":16,"fsid":"01607384-c85d-4035-8885-8f357ae44922","created":"2026-02-22T19:52:09.703535+0000","modified":"2026-02-22T19:52:27.188585+0000","last_up_change":"2026-02-22T19:52:20.171914+0000","last_in_change":"2026-02-22T19:52:10.264244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:52:21.436241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:52:23.837160+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ec0acef-b32e-4a62-a217-f9fbf183a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6808","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6809","nonce":3981027600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6810","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6811","nonce":3981027600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6814","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6815","nonce":3981027600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6812","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6813","nonce":3981027600}]},"public_addr":"10.20.193.66:6809/3981027600","cluster_addr":"10.20.193.66:6811/3981027600","heartbeat_back_addr":"10.20.193.66:6815/3981027600","heartbeat_front_addr":"10.20.193.66:6813/3981027600","state":["exists","up"]},{"osd":1,"uuid":"f352b79a-db0e-4c15-b625-23a8ab9a50b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6816","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6817","nonce":747693529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6818","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6819","nonce":747693529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6822","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6823","nonce":747693529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6820","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6821","nonce":747693529}]},"public_addr":"10.20.193.66:6817/747693529","cluster_addr":"10.20.193.66:6819/747693529","heartbeat_back_addr":"10.20.193.66:6823/747693529","heartbeat_front_addr":"10.20.193.66:6821/747693529","state":["exists","up"]},{"osd":2,"uuid":"66d84059-d5b1-41a6-9820-620d0015f5bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6800","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6801","nonce":1448960510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6802","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6803","nonce":1448960510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6806","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6807","nonce":1448960510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6804","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6805","nonce":1448960510}]},"public_addr":"10.20.193.66:6801/1448960510","cluster_addr":"10.20.193.66:6803/1448960510","heartbeat_back_addr":"10.20.193.66:6807/1448960510","heartbeat_front_addr":"10.20.193.66:6805/1448960510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.166499+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.348855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.140264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:27.708 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2026-02-22T19:52:27.708 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2026-02-22T19:52:27.709 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2026-02-22T19:52:27.790 INFO:teuthology.orchestra.run.trial066.stdout:34359738373 2026-02-22T19:52:27.790 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-22T19:52:27.794 INFO:teuthology.orchestra.run.trial066.stdout:38654705669 2026-02-22T19:52:27.794 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-22T19:52:27.984 INFO:teuthology.orchestra.run.trial066.stdout:34359738372 2026-02-22T19:52:27.990 INFO:teuthology.orchestra.run.trial066.stdout:38654705667 2026-02-22T19:52:27.995 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2026-02-22T19:52:27.997 INFO:teuthology.orchestra.run.trial066.stdout:38654705669 2026-02-22T19:52:27.997 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-22T19:52:28.002 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705667 for osd.1 2026-02-22T19:52:28.190 INFO:teuthology.orchestra.run.trial066.stdout:38654705667 2026-02-22T19:52:28.200 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705667 for osd.0 2026-02-22T19:52:28.995 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-22T19:52:29.002 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-22T19:52:29.151 INFO:teuthology.orchestra.run.trial066.stdout:34359738372 2026-02-22T19:52:29.163 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2026-02-22T19:52:29.181 INFO:teuthology.orchestra.run.trial066.stdout:38654705668 2026-02-22T19:52:29.193 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.1 2026-02-22T19:52:29.201 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-22T19:52:29.383 INFO:teuthology.orchestra.run.trial066.stdout:38654705668 2026-02-22T19:52:29.394 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.0 2026-02-22T19:52:30.163 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-22T19:52:30.193 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-22T19:52:30.313 INFO:teuthology.orchestra.run.trial066.stdout:34359738373 2026-02-22T19:52:30.324 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2026-02-22T19:52:30.325 DEBUG:teuthology.parallel:result is None 2026-02-22T19:52:30.348 INFO:teuthology.orchestra.run.trial066.stdout:38654705669 2026-02-22T19:52:30.358 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705669 for osd.1 2026-02-22T19:52:30.358 DEBUG:teuthology.parallel:result is None 2026-02-22T19:52:30.395 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-22T19:52:30.543 INFO:teuthology.orchestra.run.trial066.stdout:38654705669 2026-02-22T19:52:30.553 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705669 for osd.0 2026-02-22T19:52:30.554 DEBUG:teuthology.parallel:result is None 2026-02-22T19:52:30.554 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2026-02-22T19:52:30.554 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T19:52:30.706 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:52:30.706 INFO:teuthology.orchestra.run.trial066.stderr:dumped all 2026-02-22T19:52:30.716 INFO:teuthology.orchestra.run.trial066.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2026-02-22T19:52:29.402397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82908,"kb_used_data":1560,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283032612,"statfs":{"total":289910292480,"available":289825394688,"internally_reserved":0,"allocated":1597440,"data_stored":1294630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.212709"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.193227+0000","last_change":"2026-02-22T19:52:27.193486+0000","last_active":"2026-02-22T19:52:27.193227+0000","last_peered":"2026-02-22T19:52:27.193227+0000","last_clean":"2026-02-22T19:52:27.193227+0000","last_became_active":"2026-02-22T19:52:25.190488+0000","last_became_peered":"2026-02-22T19:52:25.190488+0000","last_unstale":"2026-02-22T19:52:27.193227+0000","last_undegraded":"2026-02-22T19:52:27.193227+0000","last_fullsized":"2026-02-22T19:52:27.193227+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:38:23.678969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091298699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.192804+0000","last_change":"2026-02-22T19:52:27.192944+0000","last_active":"2026-02-22T19:52:27.192804+0000","last_peered":"2026-02-22T19:52:27.192804+0000","last_clean":"2026-02-22T19:52:27.192804+0000","last_became_active":"2026-02-22T19:52:25.190482+0000","last_became_peered":"2026-02-22T19:52:25.190482+0000","last_unstale":"2026-02-22T19:52:27.192804+0000","last_undegraded":"2026-02-22T19:52:27.192804+0000","last_fullsized":"2026-02-22T19:52:27.192804+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:29:08.459932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037698099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.192993+0000","last_change":"2026-02-22T19:52:27.193158+0000","last_active":"2026-02-22T19:52:27.192993+0000","last_peered":"2026-02-22T19:52:27.192993+0000","last_clean":"2026-02-22T19:52:27.192993+0000","last_became_active":"2026-02-22T19:52:25.189560+0000","last_became_peered":"2026-02-22T19:52:25.189560+0000","last_unstale":"2026-02-22T19:52:27.192993+0000","last_undegraded":"2026-02-22T19:52:27.192993+0000","last_fullsized":"2026-02-22T19:52:27.192993+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:07:26.737908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059989500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.192749+0000","last_change":"2026-02-22T19:52:27.192933+0000","last_active":"2026-02-22T19:52:27.192749+0000","last_peered":"2026-02-22T19:52:27.192749+0000","last_clean":"2026-02-22T19:52:27.192749+0000","last_became_active":"2026-02-22T19:52:25.189562+0000","last_became_peered":"2026-02-22T19:52:25.189562+0000","last_unstale":"2026-02-22T19:52:27.192749+0000","last_undegraded":"2026-02-22T19:52:27.192749+0000","last_fullsized":"2026-02-22T19:52:27.192749+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:33:55.622679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038538599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.190720+0000","last_change":"2026-02-22T19:52:27.190720+0000","last_active":"2026-02-22T19:52:27.190720+0000","last_peered":"2026-02-22T19:52:27.190720+0000","last_clean":"2026-02-22T19:52:27.190720+0000","last_became_active":"2026-02-22T19:52:25.189656+0000","last_became_peered":"2026-02-22T19:52:25.189656+0000","last_unstale":"2026-02-22T19:52:27.190720+0000","last_undegraded":"2026-02-22T19:52:27.190720+0000","last_fullsized":"2026-02-22T19:52:27.190720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:44:53.529788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026044399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.516602+0000","last_change":"2026-02-22T19:52:27.516719+0000","last_active":"2026-02-22T19:52:27.516602+0000","last_peered":"2026-02-22T19:52:27.516602+0000","last_clean":"2026-02-22T19:52:27.516602+0000","last_became_active":"2026-02-22T19:52:25.189998+0000","last_became_peered":"2026-02-22T19:52:25.189998+0000","last_unstale":"2026-02-22T19:52:27.516602+0000","last_undegraded":"2026-02-22T19:52:27.516602+0000","last_fullsized":"2026-02-22T19:52:27.516602+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:34:47.988806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026744600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.516565+0000","last_change":"2026-02-22T19:52:27.516715+0000","last_active":"2026-02-22T19:52:27.516565+0000","last_peered":"2026-02-22T19:52:27.516565+0000","last_clean":"2026-02-22T19:52:27.516565+0000","last_became_active":"2026-02-22T19:52:25.189993+0000","last_became_peered":"2026-02-22T19:52:25.189993+0000","last_unstale":"2026-02-22T19:52:27.516565+0000","last_undegraded":"2026-02-22T19:52:27.516565+0000","last_fullsized":"2026-02-22T19:52:27.516565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:48:18.874126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027272700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.193047+0000","last_change":"2026-02-22T19:52:27.193332+0000","last_active":"2026-02-22T19:52:27.193047+0000","last_peered":"2026-02-22T19:52:27.193047+0000","last_clean":"2026-02-22T19:52:27.193047+0000","last_became_active":"2026-02-22T19:52:25.189445+0000","last_became_peered":"2026-02-22T19:52:25.189445+0000","last_unstale":"2026-02-22T19:52:27.193047+0000","last_undegraded":"2026-02-22T19:52:27.193047+0000","last_fullsized":"2026-02-22T19:52:27.193047+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:10:12.693851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075128700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'72","reported_seq":108,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.192381+0000","last_change":"2026-02-22T19:52:23.183055+0000","last_active":"2026-02-22T19:52:27.192381+0000","last_peered":"2026-02-22T19:52:27.192381+0000","last_clean":"2026-02-22T19:52:27.192381+0000","last_became_active":"2026-02-22T19:52:23.182893+0000","last_became_peered":"2026-02-22T19:52:23.182893+0000","last_unstale":"2026-02-22T19:52:27.192381+0000","last_undegraded":"2026-02-22T19:52:27.192381+0000","last_fullsized":"2026-02-22T19:52:27.192381+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_clean_scrub_stamp":"2026-02-22T19:52:22.177365+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:20:38.106341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":729088,"data_stored":627145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":737280,"data_stored":630716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27092,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344748,"statfs":{"total":96636764160,"available":96609021952,"internally_reserved":0,"allocated":131072,"data_stored":36769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T19:52:30.718 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T19:52:30.858 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:52:30.859 INFO:teuthology.orchestra.run.trial066.stderr:dumped all 2026-02-22T19:52:30.868 INFO:teuthology.orchestra.run.trial066.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2026-02-22T19:52:29.402397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82908,"kb_used_data":1560,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283032612,"statfs":{"total":289910292480,"available":289825394688,"internally_reserved":0,"allocated":1597440,"data_stored":1294630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.212709"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.193227+0000","last_change":"2026-02-22T19:52:27.193486+0000","last_active":"2026-02-22T19:52:27.193227+0000","last_peered":"2026-02-22T19:52:27.193227+0000","last_clean":"2026-02-22T19:52:27.193227+0000","last_became_active":"2026-02-22T19:52:25.190488+0000","last_became_peered":"2026-02-22T19:52:25.190488+0000","last_unstale":"2026-02-22T19:52:27.193227+0000","last_undegraded":"2026-02-22T19:52:27.193227+0000","last_fullsized":"2026-02-22T19:52:27.193227+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:38:23.678969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091298699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.192804+0000","last_change":"2026-02-22T19:52:27.192944+0000","last_active":"2026-02-22T19:52:27.192804+0000","last_peered":"2026-02-22T19:52:27.192804+0000","last_clean":"2026-02-22T19:52:27.192804+0000","last_became_active":"2026-02-22T19:52:25.190482+0000","last_became_peered":"2026-02-22T19:52:25.190482+0000","last_unstale":"2026-02-22T19:52:27.192804+0000","last_undegraded":"2026-02-22T19:52:27.192804+0000","last_fullsized":"2026-02-22T19:52:27.192804+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:29:08.459932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037698099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.192993+0000","last_change":"2026-02-22T19:52:27.193158+0000","last_active":"2026-02-22T19:52:27.192993+0000","last_peered":"2026-02-22T19:52:27.192993+0000","last_clean":"2026-02-22T19:52:27.192993+0000","last_became_active":"2026-02-22T19:52:25.189560+0000","last_became_peered":"2026-02-22T19:52:25.189560+0000","last_unstale":"2026-02-22T19:52:27.192993+0000","last_undegraded":"2026-02-22T19:52:27.192993+0000","last_fullsized":"2026-02-22T19:52:27.192993+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:07:26.737908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059989500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.192749+0000","last_change":"2026-02-22T19:52:27.192933+0000","last_active":"2026-02-22T19:52:27.192749+0000","last_peered":"2026-02-22T19:52:27.192749+0000","last_clean":"2026-02-22T19:52:27.192749+0000","last_became_active":"2026-02-22T19:52:25.189562+0000","last_became_peered":"2026-02-22T19:52:25.189562+0000","last_unstale":"2026-02-22T19:52:27.192749+0000","last_undegraded":"2026-02-22T19:52:27.192749+0000","last_fullsized":"2026-02-22T19:52:27.192749+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:33:55.622679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038538599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.190720+0000","last_change":"2026-02-22T19:52:27.190720+0000","last_active":"2026-02-22T19:52:27.190720+0000","last_peered":"2026-02-22T19:52:27.190720+0000","last_clean":"2026-02-22T19:52:27.190720+0000","last_became_active":"2026-02-22T19:52:25.189656+0000","last_became_peered":"2026-02-22T19:52:25.189656+0000","last_unstale":"2026-02-22T19:52:27.190720+0000","last_undegraded":"2026-02-22T19:52:27.190720+0000","last_fullsized":"2026-02-22T19:52:27.190720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:44:53.529788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026044399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.516602+0000","last_change":"2026-02-22T19:52:27.516719+0000","last_active":"2026-02-22T19:52:27.516602+0000","last_peered":"2026-02-22T19:52:27.516602+0000","last_clean":"2026-02-22T19:52:27.516602+0000","last_became_active":"2026-02-22T19:52:25.189998+0000","last_became_peered":"2026-02-22T19:52:25.189998+0000","last_unstale":"2026-02-22T19:52:27.516602+0000","last_undegraded":"2026-02-22T19:52:27.516602+0000","last_fullsized":"2026-02-22T19:52:27.516602+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:34:47.988806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026744600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.516565+0000","last_change":"2026-02-22T19:52:27.516715+0000","last_active":"2026-02-22T19:52:27.516565+0000","last_peered":"2026-02-22T19:52:27.516565+0000","last_clean":"2026-02-22T19:52:27.516565+0000","last_became_active":"2026-02-22T19:52:25.189993+0000","last_became_peered":"2026-02-22T19:52:25.189993+0000","last_unstale":"2026-02-22T19:52:27.516565+0000","last_undegraded":"2026-02-22T19:52:27.516565+0000","last_fullsized":"2026-02-22T19:52:27.516565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:48:18.874126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027272700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.193047+0000","last_change":"2026-02-22T19:52:27.193332+0000","last_active":"2026-02-22T19:52:27.193047+0000","last_peered":"2026-02-22T19:52:27.193047+0000","last_clean":"2026-02-22T19:52:27.193047+0000","last_became_active":"2026-02-22T19:52:25.189445+0000","last_became_peered":"2026-02-22T19:52:25.189445+0000","last_unstale":"2026-02-22T19:52:27.193047+0000","last_undegraded":"2026-02-22T19:52:27.193047+0000","last_fullsized":"2026-02-22T19:52:27.193047+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:10:12.693851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075128700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'72","reported_seq":108,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:27.192381+0000","last_change":"2026-02-22T19:52:23.183055+0000","last_active":"2026-02-22T19:52:27.192381+0000","last_peered":"2026-02-22T19:52:27.192381+0000","last_clean":"2026-02-22T19:52:27.192381+0000","last_became_active":"2026-02-22T19:52:23.182893+0000","last_became_peered":"2026-02-22T19:52:23.182893+0000","last_unstale":"2026-02-22T19:52:27.192381+0000","last_undegraded":"2026-02-22T19:52:27.192381+0000","last_fullsized":"2026-02-22T19:52:27.192381+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_clean_scrub_stamp":"2026-02-22T19:52:22.177365+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:20:38.106341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":729088,"data_stored":627145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":737280,"data_stored":630716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27092,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344748,"statfs":{"total":96636764160,"available":96609021952,"internally_reserved":0,"allocated":131072,"data_stored":36769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T19:52:30.870 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2026-02-22T19:52:30.870 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2026-02-22T19:52:30.870 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2026-02-22T19:52:30.870 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2026-02-22T19:52:31.026 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T19:52:31.026 INFO:teuthology.orchestra.run.trial066.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-22T19:52:31.036 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2026-02-22T19:52:31.037 INFO:teuthology.run_tasks:Running task exec... 2026-02-22T19:52:31.049 INFO:teuthology.task.exec:Executing custom commands... 2026-02-22T19:52:31.049 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@trial066.front.sepia.ceph.com 2026-02-22T19:52:31.049 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:31.231 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror] 2026-02-22T19:52:31.231 INFO:teuthology.orchestra.run.trial066.stdout: key = AQAxXptpTshCNhAAhfa9tJuc9suH/xoNvQd9ew== 2026-02-22T19:52:31.231 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror-peer" 2026-02-22T19:52:31.231 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:31.232 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror 2026-02-22T19:52:31.242 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:31.420 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.0] 2026-02-22T19:52:31.420 INFO:teuthology.orchestra.run.trial066.stdout: key = AQAxXptpYX9NOhAAzPzMz6OaTMBxtjxYrgycaQ== 2026-02-22T19:52:31.420 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:31.420 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:31.421 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror.0 2026-02-22T19:52:31.431 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:31.611 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.1] 2026-02-22T19:52:31.611 INFO:teuthology.orchestra.run.trial066.stdout: key = AQAyXptp8JTBAhAAbEpGMGHT49SrjPyn7uGqoQ== 2026-02-22T19:52:31.612 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:31.612 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:31.612 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror.1 2026-02-22T19:52:31.622 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:31.796 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.2] 2026-02-22T19:52:31.796 INFO:teuthology.orchestra.run.trial066.stdout: key = AQAyXptpGnLMBhAA86rh9ZOVFI7aL0cJK6ayXA== 2026-02-22T19:52:31.796 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:31.796 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:31.797 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror.2 2026-02-22T19:52:31.807 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:31.987 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.3] 2026-02-22T19:52:31.987 INFO:teuthology.orchestra.run.trial066.stdout: key = AQAyXptpRELfChAAV6/ogsPCZr0E+Ta8UBEp8w== 2026-02-22T19:52:31.987 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:31.987 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:31.988 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror.3 2026-02-22T19:52:31.998 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:32.172 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror] 2026-02-22T19:52:32.172 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpDLnRHhAAx4O1JZA/0bYZMH3c2pZQXA== 2026-02-22T19:52:32.172 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror-peer" 2026-02-22T19:52:32.172 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:32.173 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror 2026-02-22T19:52:32.183 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:32.360 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.0] 2026-02-22T19:52:32.360 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpdAIRIxAAFkjFfQqZwp3jWzAnNTjbzA== 2026-02-22T19:52:32.360 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:32.360 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:32.361 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror.0 2026-02-22T19:52:32.372 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:32.553 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.1] 2026-02-22T19:52:32.553 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpMRxYJxAAiT1/lMpChQKlSgF5WthFVw== 2026-02-22T19:52:32.554 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:32.554 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:32.554 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror.1 2026-02-22T19:52:32.566 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:32.743 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.2] 2026-02-22T19:52:32.743 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpoUmhKxAA6dRMvmXocaaGS3t4z0nCRA== 2026-02-22T19:52:32.743 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:32.743 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:32.743 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror.2 2026-02-22T19:52:32.755 DEBUG:teuthology.orchestra.run.trial066:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:32.935 INFO:teuthology.orchestra.run.trial066.stdout:[client.mirror.3] 2026-02-22T19:52:32.936 INFO:teuthology.orchestra.run.trial066.stdout: key = AQBYXptpx6XlLxAAYqPm5YyrChSCiW1T/JEvbg== 2026-02-22T19:52:32.936 INFO:teuthology.orchestra.run.trial066.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:32.936 INFO:teuthology.orchestra.run.trial066.stdout: caps osd = "profile rbd" 2026-02-22T19:52:32.936 INFO:teuthology.orchestra.run.trial066.stderr:updated caps for client.mirror.3 2026-02-22T19:52:32.946 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-22T19:52:32.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-22T19:52:32.964 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-22T19:52:32.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-22T19:52:32.966 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-22T19:52:32.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-22T19:52:32.976 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-22T19:52:32.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-22T19:52:32.977 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-22T19:52:33.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2026-02-22T19:52:33.001 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2026-02-22T19:52:33.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2026-02-22T19:52:33.002 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-22T19:52:33.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2026-02-22T19:52:33.020 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2026-02-22T19:52:33.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2026-02-22T19:52:33.022 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-22T19:52:33.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-22T19:52:33.039 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-22T19:52:33.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-22T19:52:33.041 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-22T19:52:33.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-22T19:52:33.056 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-22T19:52:33.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-22T19:52:33.058 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-22T19:52:33.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2026-02-22T19:52:33.075 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2026-02-22T19:52:33.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2026-02-22T19:52:33.077 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-22T19:52:33.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2026-02-22T19:52:33.093 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2026-02-22T19:52:33.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2026-02-22T19:52:33.095 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2026-02-22T19:52:33.114 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1771789953 2026-02-22T19:52:33.115 DEBUG:tasks.rbd_mirror_thrash:Yielding 2026-02-22T19:52:33.115 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2026-02-22T19:52:33.128 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1771789953 2026-02-22T19:52:33.129 DEBUG:tasks.rbd_mirror_thrash:Yielding 2026-02-22T19:52:33.129 INFO:teuthology.run_tasks:Running task workunit... 2026-02-22T19:52:33.149 INFO:tasks.workunit:Pulling workunits from ref f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:52:33.150 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-22T19:52:33.150 DEBUG:teuthology.orchestra.run.trial066:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-22T19:52:33.150 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2026-02-22T19:52:33.150 INFO:tasks.rbd_mirror_thrash:waiting for 77.49141613925698 secs before thrashing 2026-02-22T19:52:33.150 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2026-02-22T19:52:33.151 INFO:tasks.rbd_mirror_thrash:waiting for 106.52515796862903 secs before thrashing 2026-02-22T19:52:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:52:33.155 INFO:teuthology.orchestra.run.trial066.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2026-02-22T19:52:33.155 DEBUG:teuthology.orchestra.run.trial066:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-22T19:52:33.201 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-22T19:52:33.201 DEBUG:teuthology.orchestra.run.trial066:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2026-02-22T19:52:33.246 INFO:tasks.workunit:timeout=6h 2026-02-22T19:52:33.246 INFO:tasks.workunit:cleanup=True 2026-02-22T19:52:33.247 DEBUG:teuthology.orchestra.run.trial066:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:52:33.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2026-02-22T19:52:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Note: switching to 'f90f4eaebb28b619cf76e80cbbe997656da69579'. 2026-02-22T19:52:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: 2026-02-22T19:52:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-22T19:52:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:state without impacting any branches by switching back to a branch. 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: git switch -c 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Or undo this operation with: 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: git switch - 2026-02-22T19:52:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: 2026-02-22T19:52:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-22T19:52:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: 2026-02-22T19:52:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:HEAD is now at f90f4eaebb2 Merge branch 'wip-74712-squid' of https://github.com/idryomov/ceph into wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T19:52:55.450 DEBUG:teuthology.orchestra.run.trial066:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2026-02-22T19:52:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-22T19:52:55.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2026-02-22T19:52:55.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-22T19:52:55.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-22T19:52:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-22T19:52:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2026-02-22T19:52:55.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2026-02-22T19:52:55.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-22T19:52:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2026-02-22T19:52:55.680 DEBUG:teuthology.orchestra.run.trial066:> set -ex 2026-02-22T19:52:55.680 DEBUG:teuthology.orchestra.run.trial066:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2026-02-22T19:52:55.726 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2026-02-22T19:52:55.729 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2026-02-22T19:52:55.729 DEBUG:teuthology.orchestra.run.trial066:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=pool RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=journal RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2026-02-22T19:52:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ IMAGE_COUNT=50 2026-02-22T19:52:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ export LOCKDEP=0 2026-02-22T19:52:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ LOCKDEP=0 2026-02-22T19:52:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2026-02-22T19:52:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2026-02-22T19:52:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ type xmlstarlet 2026-02-22T19:52:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ XMLSTARLET=xmlstarlet 2026-02-22T19:52:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ RBD_MIRROR_INSTANCES=4 2026-02-22T19:52:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CLUSTER1=cluster1 2026-02-22T19:52:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CLUSTER2=cluster2 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ PEER_CLUSTER_SUFFIX= 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ POOL=mirror 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ PARENT_POOL=mirror_parent 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ NS1=ns1 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ NS2=ns2 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ TEMPDIR= 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ID=mirror 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ RBD_MIRROR_MODE=journal 2026-02-22T19:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ MIRROR_POOL_MODE=pool 2026-02-22T19:52:55.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' journal = snapshot ']' 2026-02-22T19:52:55.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ export 'CEPH_ARGS=--id mirror' 2026-02-22T19:52:55.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS='--id mirror' 2026-02-22T19:52:55.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ LAST_MIRROR_INSTANCE=3 2026-02-22T19:52:55.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2026-02-22T19:52:55.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2026-02-22T19:52:55.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2026-02-22T19:52:55.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_BIN=. 2026-02-22T19:52:55.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_SRC=. 2026-02-22T19:52:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' -e CMakeCache.txt ']' 2026-02-22T19:52:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 -gt 0 ']' 2026-02-22T19:52:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ setup 2026-02-22T19:52:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local c 2026-02-22T19:52:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ trap 'cleanup $?' INT TERM EXIT 2026-02-22T19:52:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ setup_tempdir 2026-02-22T19:52:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:52:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ mktemp -d 2026-02-22T19:52:55.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ TEMPDIR=/tmp/tmp.Xb7Kt1LMAh 2026-02-22T19:52:55.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -z 1 ']' 2026-02-22T19:52:55.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ setup_pools cluster1 cluster2 2026-02-22T19:52:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:52:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local remote_cluster=cluster2 2026-02-22T19:52:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mon_map_file 2026-02-22T19:52:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mon_addr 2026-02-22T19:52:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local admin_key_file 2026-02-22T19:52:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local uuid 2026-02-22T19:52:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:52:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2026-02-22T19:52:56.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:pool 'mirror' already exists 2026-02-22T19:52:56.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:52:56.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2026-02-22T19:52:57.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:pool 'mirror_parent' already exists 2026-02-22T19:52:57.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:52:57.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 pool init mirror 2026-02-22T19:53:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:53:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2026-02-22T19:53:03.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:03.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:53:03.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2026-02-22T19:53:03.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2026-02-22T19:53:03.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2026-02-22T19:53:03.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2026-02-22T19:53:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 namespace create mirror_parent/ns1 2026-02-22T19:53:03.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2026-02-22T19:53:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2026-02-22T19:53:03.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent/ns1 pool 2026-02-22T19:53:03.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -z ']' 2026-02-22T19:53:03.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -z ']' 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ peer_add cluster1 mirror cluster2 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local client_cluster=cluster2 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local remote_cluster=cluster2 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local uuid_var_name 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local error_code 2026-02-22T19:53:03.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local peer_uuid 2026-02-22T19:53:03.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 16 32 2026-02-22T19:53:03.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T19:53:03.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2026-02-22T19:53:03.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ peer_uuid=c3484687-9bd4-465e-890b-04686b65eee6 2026-02-22T19:53:03.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T19:53:03.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T19:53:03.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 17 ']' 2026-02-22T19:53:03.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 -eq 0 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ peer_add cluster1 mirror_parent cluster2 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror_parent 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local client_cluster=cluster2 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local remote_cluster=cluster2 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local uuid_var_name 2026-02-22T19:53:03.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:03.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local error_code 2026-02-22T19:53:03.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local peer_uuid 2026-02-22T19:53:03.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 16 32 2026-02-22T19:53:03.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T19:53:03.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2026-02-22T19:53:03.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ peer_uuid=383d2e66-91ad-448c-a93b-af3981311987 2026-02-22T19:53:03.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T19:53:03.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T19:53:03.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 17 ']' 2026-02-22T19:53:03.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 -eq 0 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ setup_pools cluster2 cluster1 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local remote_cluster=cluster1 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mon_map_file 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mon_addr 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local admin_key_file 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local uuid 2026-02-22T19:53:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:53:03.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2026-02-22T19:53:04.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:pool 'mirror' already exists 2026-02-22T19:53:04.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:53:04.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2026-02-22T19:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:pool 'mirror_parent' already exists 2026-02-22T19:53:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:53:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 pool init mirror 2026-02-22T19:53:08.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:53:08.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2026-02-22T19:53:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T19:53:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2026-02-22T19:53:11.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2026-02-22T19:53:11.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2026-02-22T19:53:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2026-02-22T19:53:12.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 namespace create mirror_parent/ns1 2026-02-22T19:53:12.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2026-02-22T19:53:12.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2026-02-22T19:53:12.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent/ns1 pool 2026-02-22T19:53:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -z ']' 2026-02-22T19:53:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -z ']' 2026-02-22T19:53:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ peer_add cluster2 mirror cluster1 2026-02-22T19:53:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:53:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local client_cluster=cluster1 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local remote_cluster=cluster1 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local uuid_var_name 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local error_code 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local peer_uuid 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 16 32 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T19:53:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2026-02-22T19:53:12.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ peer_uuid=cc34e230-b1bd-44b6-b08b-75dee4295807 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 17 ']' 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 -eq 0 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ peer_add cluster2 mirror_parent cluster1 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror_parent 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local client_cluster=cluster1 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local remote_cluster=cluster1 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local uuid_var_name 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local error_code 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local peer_uuid 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 16 32 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T19:53:12.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2026-02-22T19:53:12.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ peer_uuid=e89deac5-2d3b-4d3a-934b-6eb3fd9ea00b 2026-02-22T19:53:12.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T19:53:12.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T19:53:12.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 17 ']' 2026-02-22T19:53:12.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 -eq 0 2026-02-22T19:53:12.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:12.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:53:12.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:12.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirrors cluster1 2026-02-22T19:53:12.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:53:12.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:53:12.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:12.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirror cluster1:0 2026-02-22T19:53:12.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:12.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:53:12.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:12.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T19:53:12.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirror cluster1:1 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirror cluster1:2 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:53:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirror cluster1:3 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:53:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T19:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirrors cluster2 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirror cluster2:0 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2:0 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster2:0 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster2 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:53:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster2 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster2 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirror cluster2:1 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2:1 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster2:1 2026-02-22T19:53:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster2 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster2 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster2 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirror cluster2:2 2026-02-22T19:53:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2:2 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster2:2 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster2 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster2 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster2 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirror cluster2:3 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2:3 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster2:3 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster2 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster2 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster2 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ tee -a /tmp/tmp.Xb7Kt1LMAh/rbd-mirror.test.log 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ date '+%F %T' 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo 2026-02-22 19:53:12 TEST: add image and test replay after client crashes 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:2026-02-22 19:53:12 TEST: add image and test replay after client crashes 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=test 2026-02-22T19:53:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 512M 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror test 512M 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:12.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:12.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T19:53:12.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T19:53:12.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T19:53:12.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:53:12.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/test 2026-02-22T19:53:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T19:53:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T19:53:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T19:53:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T19:53:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:53:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:53:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:12.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:53:12.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:53:12.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:12.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:12.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T19:53:12.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:53:12.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:12.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:53:13.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:53:13.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:53:13.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T19:53:13.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:13.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:53:13.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:13.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:53:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:53:13.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:13.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:13.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:13.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:13.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:13.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:13.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:13.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:13.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:13.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:13.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:13.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:13.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T19:53:13.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T19:53:13.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:13.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:13.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:13.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:13.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T19:53:13.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T19:53:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:13.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:13.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T19:53:13.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T19:53:13.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:13.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:13.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:13.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T19:53:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T19:53:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T19:53:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T19:53:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:13.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:13.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:13.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:13.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T19:53:13.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:13.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:13.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:13.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:13.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 1 2026-02-22T19:53:14.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:14.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:53:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:53:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:14.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:14.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:14.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:14.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:14.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:14.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:14.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T19:53:14.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T19:53:14.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:14.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:14.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:14.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:14.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:14.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:14.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:14.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:14.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:14.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:14.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:14.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:14.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T19:53:14.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T19:53:14.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:14.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:14.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:14.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:14.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T19:53:14.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:14.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:14.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T19:53:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T19:53:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T19:53:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T19:53:14.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:14.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:14.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:14.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:14.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T19:53:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:14.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:14.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 2 2026-02-22T19:53:16.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:16.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:53:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:53:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:53:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:53:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:53:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:16.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:16.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:16.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:16.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T19:53:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:16.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:16.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:16.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:16.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:16.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:16.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:16.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T19:53:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T19:53:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T19:53:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T19:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T19:53:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T19:53:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T19:53:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T19:53:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:16.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:16.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:16.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:16.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T19:53:17.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:17.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:17.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:17.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:17.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 4 2026-02-22T19:53:21.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:21.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:53:21.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:21.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:21.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:21.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:21.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:21.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:53:21.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:53:21.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:21.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:21.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:21.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:21.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:21.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:21.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:21.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:21.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T19:53:21.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:21.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:21.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:21.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:21.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:21.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T19:53:21.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T19:53:21.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:21.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T19:53:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T19:53:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:21.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T19:53:21.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:21.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:21.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:21.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:21.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:21.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T19:53:21.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T19:53:21.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:21.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:21.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:21.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:21.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T19:53:21.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:21.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:21.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:21.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 8 2026-02-22T19:53:29.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:29.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:53:29.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:29.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:29.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:29.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:29.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:29.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T19:53:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T19:53:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T19:53:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T19:53:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:29.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:29.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:29.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:29.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:29.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:29.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:29.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:29.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:29.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T19:53:29.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T19:53:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T19:53:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T19:53:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:29.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:29.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:29.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:29.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T19:53:29.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:29.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:29.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T19:53:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T19:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T19:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T19:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:29.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:29.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:29.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:29.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T19:53:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T19:53:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:29.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 8 2026-02-22T19:53:37.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:37.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:53:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:53:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:53:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:53:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:53:37.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:53:37.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:37.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:37.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:37.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:37.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T19:53:37.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:37.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:37.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T19:53:37.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T19:53:37.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T19:53:37.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T19:53:37.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:53:37.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name= 2026-02-22T19:53:37.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 10 2026-02-22T19:53:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T19:53:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T19:53:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:53:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T19:53:37.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=14 2026-02-22T19:53:37.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T19:53:37.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T19:53:38.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 75736064 bytes 2026-02-22T19:53:39.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 189446144 bytes 2026-02-22T19:53:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 250429440 bytes 2026-02-22T19:53:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 294493184 bytes 2026-02-22T19:53:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 352947200 bytes 2026-02-22T19:53:42.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 418611200 bytes 2026-02-22T19:53:43.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 483358720 bytes 2026-02-22T19:53:43.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 503226368 bytes 2026-02-22T19:53:43.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 503835648 bytes 2026-02-22T19:53:44.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 589334528 bytes 2026-02-22T19:53:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 590442496 bytes 2026-02-22T19:53:44.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 612722688 bytes 2026-02-22T19:53:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 677078016 bytes 2026-02-22T19:53:45.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 695392256 bytes 2026-02-22T19:53:46.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 742866944 bytes 2026-02-22T19:53:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 746276864 bytes 2026-02-22T19:53:46.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1700 IOs, 765990912 bytes 2026-02-22T19:53:46.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1800 IOs, 790895616 bytes 2026-02-22T19:53:47.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1900 IOs, 799057920 bytes 2026-02-22T19:53:47.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2000 IOs, 861917184 bytes 2026-02-22T19:53:48.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2100 IOs, 906384384 bytes 2026-02-22T19:53:48.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2200 IOs, 951729152 bytes 2026-02-22T19:53:49.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2300 IOs, 1012120576 bytes 2026-02-22T19:53:50.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2400 IOs, 1073217536 bytes 2026-02-22T19:53:50.643 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5589436026059355) > weight (0.25) 2026-02-22T19:53:50.643 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7577477828695919) > weight (0.25) 2026-02-22T19:53:50.643 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5215367065640535) > weight (0.25) 2026-02-22T19:53:50.643 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.48489216985384975) > weight (0.25) 2026-02-22T19:53:50.643 INFO:tasks.rbd_mirror_thrash:waiting for 85.31607832834017 secs before thrashing 2026-02-22T19:53:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2500 IOs, 1132470272 bytes 2026-02-22T19:53:51.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T19:53:51.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T19:53:51.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:51.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:53:52.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:53:52.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:53:52.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:52.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:52.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:52.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:53:52.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:53:52.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:53:52.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:53:52.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:53:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T19:53:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:53:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T19:53:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-22T19:53:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.0 on trial066 2026-02-22T19:53:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:53:36 2026-02-22T19:53:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T19:53:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T19:53:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T19:53:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T19:53:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:53:35' 2026-02-22T19:53:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:52.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T19:53:52.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T19:53:52.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-22T19:53:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:53:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:53:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T19:53:52.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:53:52.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:52.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.0 on trial066 2026-02-22T19:53:52.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T19:53:52.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:53:36' 2026-02-22T19:53:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:53:36' 2026-02-22T19:53:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:53:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 19:53:36' 2026-02-22T19:53:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:53:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:53:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:52.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142024257mirror.0trial06619.2.3-689-gf90f4eaetrueOK42064260mirror.1trial06619.2.3-689-gf90f4eaefalseOK42104263mirror.2trial06619.2.3-689-gf90f4eaefalseOK42134266mirror.3trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42024257mirror.0trial0662026-02-22 19:53:36cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 19:53:35' 2026-02-22T19:53:52.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T19:53:52.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:53:52.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:53:36' 2026-02-22T19:53:52.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:53:52.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T19:53:52.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:53:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T19:53:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:53:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:53:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:53:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap1 2026-02-22T19:53:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap1 2026-02-22T19:53:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:53:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:53:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:53:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap1 2026-02-22T19:53:52.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2026-02-22T19:54:19.676 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-22T19:54:19.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-22T19:54:19.677 INFO:tasks.rbd_mirror_thrash:waiting for 6.8464383280550365 secs before reviving daemons 2026-02-22T19:54:19.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T19:54:19.671+0000 7f9bdbef5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 20056) UID: 1000 2026-02-22T19:54:24.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:54:24.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:54:25.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:25.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:25.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T19:54:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T19:54:25.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:25.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T19:54:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:54:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:54:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:54:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:25.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T19:54:25.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:25.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:25.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T19:54:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T19:54:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T19:54:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T19:54:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T19:54:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T19:54:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:26.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:26.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:26.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:26.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:26.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:26.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:26.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:26.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:26.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:26.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:26.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:26.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:26.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=36321], [object_number=32, tag_tid=2, entry_tid=36320], [object_number=35, tag_tid=2, entry_tid=36319], [object_number=34, tag_tid=2, entry_tid=36318]]], state=connected] 2026-02-22T19:54:26.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:26.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:26.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:26.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:26.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:26.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:26.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:26.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:26.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:26.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=36321], [object_number=32, tag_tid=2, entry_tid=36320], [object_number=35, tag_tid=2, entry_tid=36319], [object_number=34, tag_tid=2, entry_tid=36318]]], state=connected] 2026-02-22T19:54:26.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=36321' 2026-02-22T19:54:26.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=36321' 2026-02-22T19:54:26.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=2, entry_tid=36321' '!=' '' 2026-02-22T19:54:26.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:26.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=2, entry_tid=36321' = '' 2026-02-22T19:54:26.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=33, tag_tid=2, entry_tid=36321' 2026-02-22T19:54:26.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:26.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:26.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:26.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=36321' 2026-02-22T19:54:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:26.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:26.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:26.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=36321' 2026-02-22T19:54:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36321 2026-02-22T19:54:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 36321 2026-02-22T19:54:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:26.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:26.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:26.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:26.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:26.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:26.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:26.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:26.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:26.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:26.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:26.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:26.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:26.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:26.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:26.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:26.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:26.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:26.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:26.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:26.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:26.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:26.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:26.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:26.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:26.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:26.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:26.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:26.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:26.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:26.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:26.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:26.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:26.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:26.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:26.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:26.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:26.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:26.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:26.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:26.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=36609], [object_number=32, tag_tid=2, entry_tid=36608], [object_number=35, tag_tid=2, entry_tid=36607], [object_number=34, tag_tid=2, entry_tid=36606]]], state=connected] 2026-02-22T19:54:26.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:26.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:26.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:26.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:26.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:26.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.524 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-22T19:54:26.525 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T19:54:26.525 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-22T19:54:26.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-22T19:54:26.525 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-22T19:54:26.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-22T19:54:26.527 INFO:tasks.rbd_mirror_thrash:waiting for 101.70449836272516 secs before thrashing 2026-02-22T19:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=36609], [object_number=32, tag_tid=2, entry_tid=36608], [object_number=35, tag_tid=2, entry_tid=36607], [object_number=34, tag_tid=2, entry_tid=36606]]], state=connected] 2026-02-22T19:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=36609' 2026-02-22T19:54:26.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=36609' 2026-02-22T19:54:26.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=2, entry_tid=36609' '!=' 'object_number=33, tag_tid=2, entry_tid=36321' 2026-02-22T19:54:26.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:26.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=2, entry_tid=36609' = 'object_number=33, tag_tid=2, entry_tid=36321' 2026-02-22T19:54:26.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=33, tag_tid=2, entry_tid=36609' 2026-02-22T19:54:26.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:26.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:26.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:26.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=36609' 2026-02-22T19:54:26.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:26.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:26.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:26.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:26.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:26.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=36609' 2026-02-22T19:54:26.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:26.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36609 2026-02-22T19:54:26.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 36609 2026-02-22T19:54:26.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:26.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:26.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:26.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:26.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:26.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:26.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:26.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:26.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:26.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:26.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:26.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:26.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:26.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:26.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:26.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:26.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:26.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:26.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:26.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:26.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:26.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:26.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:26.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:26.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:26.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:26.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:26.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:26.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:26.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:26.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:26.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:26.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:26.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:26.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=36894], [object_number=33, tag_tid=2, entry_tid=36893], [object_number=32, tag_tid=2, entry_tid=36892], [object_number=35, tag_tid=2, entry_tid=36891]]], state=connected] 2026-02-22T19:54:26.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:26.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:26.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:26.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:26.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:26.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=36894], [object_number=33, tag_tid=2, entry_tid=36893], [object_number=32, tag_tid=2, entry_tid=36892], [object_number=35, tag_tid=2, entry_tid=36891]]], state=connected] 2026-02-22T19:54:26.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=34, tag_tid=2, entry_tid=36894' 2026-02-22T19:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=34, tag_tid=2, entry_tid=36894' 2026-02-22T19:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=36894' '!=' 'object_number=33, tag_tid=2, entry_tid=36609' 2026-02-22T19:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=36894' = 'object_number=33, tag_tid=2, entry_tid=36609' 2026-02-22T19:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=34, tag_tid=2, entry_tid=36894' 2026-02-22T19:54:26.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:26.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:26.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:26.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=36894' 2026-02-22T19:54:26.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:26.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:26.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:26.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:26.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:26.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=36894' 2026-02-22T19:54:26.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:26.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36894 2026-02-22T19:54:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 36894 2026-02-22T19:54:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:27.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:27.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:27.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:27.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:27.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:27.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:27.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=2, entry_tid=37143], [object_number=34, tag_tid=2, entry_tid=37142], [object_number=33, tag_tid=2, entry_tid=37141], [object_number=32, tag_tid=2, entry_tid=37140]]], state=connected] 2026-02-22T19:54:27.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:27.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:27.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:27.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:27.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=2, entry_tid=37143], [object_number=34, tag_tid=2, entry_tid=37142], [object_number=33, tag_tid=2, entry_tid=37141], [object_number=32, tag_tid=2, entry_tid=37140]]], state=connected] 2026-02-22T19:54:27.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=35, tag_tid=2, entry_tid=37143' 2026-02-22T19:54:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=35, tag_tid=2, entry_tid=37143' 2026-02-22T19:54:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=35, tag_tid=2, entry_tid=37143' '!=' 'object_number=34, tag_tid=2, entry_tid=36894' 2026-02-22T19:54:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:27.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=35, tag_tid=2, entry_tid=37143' = 'object_number=34, tag_tid=2, entry_tid=36894' 2026-02-22T19:54:27.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=35, tag_tid=2, entry_tid=37143' 2026-02-22T19:54:27.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:27.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:27.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:27.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=35, tag_tid=2, entry_tid=37143' 2026-02-22T19:54:27.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:27.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:27.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:27.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:27.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:27.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=35, tag_tid=2, entry_tid=37143' 2026-02-22T19:54:27.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:27.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37143 2026-02-22T19:54:27.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 37143 2026-02-22T19:54:27.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:27.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:27.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:27.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:27.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:27.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:27.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:27.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:27.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:27.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:27.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:27.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:27.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:27.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:27.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:27.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:27.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:27.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:27.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:27.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:27.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:27.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:27.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:27.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:27.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:27.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:27.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:27.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:27.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:27.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:27.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:27.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:27.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=2, entry_tid=37403], [object_number=34, tag_tid=2, entry_tid=37402], [object_number=33, tag_tid=2, entry_tid=37401], [object_number=32, tag_tid=2, entry_tid=37400]]], state=connected] 2026-02-22T19:54:27.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:27.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:27.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:27.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:27.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:27.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:27.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T19:54:27.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:27.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:27.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:27.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=2, entry_tid=37403], [object_number=34, tag_tid=2, entry_tid=37402], [object_number=33, tag_tid=2, entry_tid=37401], [object_number=32, tag_tid=2, entry_tid=37400]]], state=connected] 2026-02-22T19:54:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=35, tag_tid=2, entry_tid=37403' 2026-02-22T19:54:27.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=35, tag_tid=2, entry_tid=37403' 2026-02-22T19:54:27.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=35, tag_tid=2, entry_tid=37403' '!=' 'object_number=35, tag_tid=2, entry_tid=37143' 2026-02-22T19:54:27.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:27.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=35, tag_tid=2, entry_tid=37403' = 'object_number=35, tag_tid=2, entry_tid=37143' 2026-02-22T19:54:27.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=35, tag_tid=2, entry_tid=37403' 2026-02-22T19:54:27.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:27.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:27.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:27.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=35, tag_tid=2, entry_tid=37403' 2026-02-22T19:54:27.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:27.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:27.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:27.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:27.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=35, tag_tid=2, entry_tid=37403' 2026-02-22T19:54:27.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:27.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37403 2026-02-22T19:54:27.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 37403 2026-02-22T19:54:27.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:27.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:27.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:27.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:27.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:27.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:27.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:27.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:27.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:27.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:27.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:27.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:27.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:27.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:27.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:27.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:27.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:27.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:27.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:27.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:27.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:27.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:27.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:27.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:27.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:27.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:27.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:27.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=37543], [object_number=38, tag_tid=2, entry_tid=37542], [object_number=37, tag_tid=2, entry_tid=37541], [object_number=36, tag_tid=2, entry_tid=37540]]], state=connected] 2026-02-22T19:54:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:27.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:27.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:27.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:27.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=37543], [object_number=38, tag_tid=2, entry_tid=37542], [object_number=37, tag_tid=2, entry_tid=37541], [object_number=36, tag_tid=2, entry_tid=37540]]], state=connected] 2026-02-22T19:54:27.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:27.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=39, tag_tid=2, entry_tid=37543' 2026-02-22T19:54:27.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=39, tag_tid=2, entry_tid=37543' 2026-02-22T19:54:27.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=37543' '!=' 'object_number=35, tag_tid=2, entry_tid=37403' 2026-02-22T19:54:27.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:27.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=37543' = 'object_number=35, tag_tid=2, entry_tid=37403' 2026-02-22T19:54:27.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=39, tag_tid=2, entry_tid=37543' 2026-02-22T19:54:27.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:27.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:27.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=37543' 2026-02-22T19:54:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:27.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:27.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:28.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:28.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:28.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:28.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=37543' 2026-02-22T19:54:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37543 2026-02-22T19:54:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 37543 2026-02-22T19:54:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:28.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:28.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:28.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:28.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:28.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:28.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:28.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:28.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:28.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:28.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:28.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:28.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=37794], [object_number=37, tag_tid=2, entry_tid=37793], [object_number=36, tag_tid=2, entry_tid=37792], [object_number=39, tag_tid=2, entry_tid=37791]]], state=connected] 2026-02-22T19:54:28.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:28.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:28.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:28.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:28.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:28.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:28.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:28.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:28.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:28.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:28.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:28.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:28.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=37794], [object_number=37, tag_tid=2, entry_tid=37793], [object_number=36, tag_tid=2, entry_tid=37792], [object_number=39, tag_tid=2, entry_tid=37791]]], state=connected] 2026-02-22T19:54:28.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=2, entry_tid=37794' 2026-02-22T19:54:28.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=38, tag_tid=2, entry_tid=37794' 2026-02-22T19:54:28.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=37794' '!=' 'object_number=39, tag_tid=2, entry_tid=37543' 2026-02-22T19:54:28.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:28.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=37794' = 'object_number=39, tag_tid=2, entry_tid=37543' 2026-02-22T19:54:28.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=38, tag_tid=2, entry_tid=37794' 2026-02-22T19:54:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=37794' 2026-02-22T19:54:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=37794' 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37794 2026-02-22T19:54:28.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 37794 2026-02-22T19:54:28.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:28.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:28.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:28.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:28.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:28.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:28.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:28.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:28.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:28.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:28.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:28.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:28.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:28.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:28.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:28.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:28.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:28.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:28.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:28.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:28.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:28.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:28.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=2, entry_tid=38108], [object_number=39, tag_tid=2, entry_tid=38107], [object_number=38, tag_tid=2, entry_tid=38106], [object_number=37, tag_tid=2, entry_tid=38105]]], state=connected] 2026-02-22T19:54:28.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:28.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:28.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:28.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:28.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:28.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:28.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:28.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:28.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:28.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:28.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=2, entry_tid=38108], [object_number=39, tag_tid=2, entry_tid=38107], [object_number=38, tag_tid=2, entry_tid=38106], [object_number=37, tag_tid=2, entry_tid=38105]]], state=connected] 2026-02-22T19:54:28.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:28.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=36, tag_tid=2, entry_tid=38108' 2026-02-22T19:54:28.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=36, tag_tid=2, entry_tid=38108' 2026-02-22T19:54:28.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=2, entry_tid=38108' '!=' 'object_number=38, tag_tid=2, entry_tid=37794' 2026-02-22T19:54:28.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:28.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=2, entry_tid=38108' = 'object_number=38, tag_tid=2, entry_tid=37794' 2026-02-22T19:54:28.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=36, tag_tid=2, entry_tid=38108' 2026-02-22T19:54:28.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:28.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:28.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:28.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=2, entry_tid=38108' 2026-02-22T19:54:28.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:28.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:28.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:28.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:28.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:28.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=2, entry_tid=38108' 2026-02-22T19:54:28.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:28.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:28.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38108 2026-02-22T19:54:28.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 38108 2026-02-22T19:54:28.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:28.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:28.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:28.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:28.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:28.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:29.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:29.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:29.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:29.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:29.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:29.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:29.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:29.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:29.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=38407], [object_number=38, tag_tid=2, entry_tid=38406], [object_number=37, tag_tid=2, entry_tid=38405], [object_number=36, tag_tid=2, entry_tid=38404]]], state=connected] 2026-02-22T19:54:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:29.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:29.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:29.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:29.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:29.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:29.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:29.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:29.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=38407], [object_number=38, tag_tid=2, entry_tid=38406], [object_number=37, tag_tid=2, entry_tid=38405], [object_number=36, tag_tid=2, entry_tid=38404]]], state=connected] 2026-02-22T19:54:29.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=39, tag_tid=2, entry_tid=38407' 2026-02-22T19:54:29.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=39, tag_tid=2, entry_tid=38407' 2026-02-22T19:54:29.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=38407' '!=' 'object_number=36, tag_tid=2, entry_tid=38108' 2026-02-22T19:54:29.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:29.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=38407' = 'object_number=36, tag_tid=2, entry_tid=38108' 2026-02-22T19:54:29.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=39, tag_tid=2, entry_tid=38407' 2026-02-22T19:54:29.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:29.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:29.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:29.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=38407' 2026-02-22T19:54:29.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:29.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:29.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:29.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:29.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=38407' 2026-02-22T19:54:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38407 2026-02-22T19:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 38407 2026-02-22T19:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:29.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:29.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:29.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:29.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:29.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:29.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:29.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=38671], [object_number=38, tag_tid=2, entry_tid=38670], [object_number=37, tag_tid=2, entry_tid=38669], [object_number=36, tag_tid=2, entry_tid=38668]]], state=connected] 2026-02-22T19:54:29.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:29.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:29.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:29.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:29.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=38671], [object_number=38, tag_tid=2, entry_tid=38670], [object_number=37, tag_tid=2, entry_tid=38669], [object_number=36, tag_tid=2, entry_tid=38668]]], state=connected] 2026-02-22T19:54:29.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=39, tag_tid=2, entry_tid=38671' 2026-02-22T19:54:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=39, tag_tid=2, entry_tid=38671' 2026-02-22T19:54:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=38671' '!=' 'object_number=39, tag_tid=2, entry_tid=38407' 2026-02-22T19:54:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=38671' = 'object_number=39, tag_tid=2, entry_tid=38407' 2026-02-22T19:54:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=39, tag_tid=2, entry_tid=38671' 2026-02-22T19:54:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:29.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:29.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=38671' 2026-02-22T19:54:29.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:29.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:29.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=38671' 2026-02-22T19:54:29.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:29.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38671 2026-02-22T19:54:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 38671 2026-02-22T19:54:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:29.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:29.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:29.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:29.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:29.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:29.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:29.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:29.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:29.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:29.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:29.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:29.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:29.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=38929], [object_number=36, tag_tid=2, entry_tid=38928], [object_number=39, tag_tid=2, entry_tid=38927], [object_number=38, tag_tid=2, entry_tid=38926]]], state=connected] 2026-02-22T19:54:29.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:29.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:29.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:29.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:29.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:29.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:29.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:29.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:29.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:29.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:29.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:29.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:29.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:29.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:29.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:29.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=38929], [object_number=36, tag_tid=2, entry_tid=38928], [object_number=39, tag_tid=2, entry_tid=38927], [object_number=38, tag_tid=2, entry_tid=38926]]], state=connected] 2026-02-22T19:54:29.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=37, tag_tid=2, entry_tid=38929' 2026-02-22T19:54:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=37, tag_tid=2, entry_tid=38929' 2026-02-22T19:54:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=37, tag_tid=2, entry_tid=38929' '!=' 'object_number=39, tag_tid=2, entry_tid=38671' 2026-02-22T19:54:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=37, tag_tid=2, entry_tid=38929' = 'object_number=39, tag_tid=2, entry_tid=38671' 2026-02-22T19:54:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=37, tag_tid=2, entry_tid=38929' 2026-02-22T19:54:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:29.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=38929' 2026-02-22T19:54:29.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:29.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:29.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=38929' 2026-02-22T19:54:29.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:29.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38929 2026-02-22T19:54:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 38929 2026-02-22T19:54:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:30.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:30.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:30.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:30.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:30.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:30.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:30.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:30.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:30.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:30.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:30.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:30.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:30.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:30.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:30.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:30.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:30.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:30.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=2, entry_tid=39196], [object_number=39, tag_tid=2, entry_tid=39195], [object_number=38, tag_tid=2, entry_tid=39194], [object_number=37, tag_tid=2, entry_tid=39193]]], state=connected] 2026-02-22T19:54:30.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:30.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:30.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:30.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:30.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:30.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:30.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:30.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:30.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=2, entry_tid=39196], [object_number=39, tag_tid=2, entry_tid=39195], [object_number=38, tag_tid=2, entry_tid=39194], [object_number=37, tag_tid=2, entry_tid=39193]]], state=connected] 2026-02-22T19:54:30.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=36, tag_tid=2, entry_tid=39196' 2026-02-22T19:54:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=36, tag_tid=2, entry_tid=39196' 2026-02-22T19:54:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=2, entry_tid=39196' '!=' 'object_number=37, tag_tid=2, entry_tid=38929' 2026-02-22T19:54:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=2, entry_tid=39196' = 'object_number=37, tag_tid=2, entry_tid=38929' 2026-02-22T19:54:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=36, tag_tid=2, entry_tid=39196' 2026-02-22T19:54:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:30.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=2, entry_tid=39196' 2026-02-22T19:54:30.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:30.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:30.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:30.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:30.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:30.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=2, entry_tid=39196' 2026-02-22T19:54:30.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:30.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39196 2026-02-22T19:54:30.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 39196 2026-02-22T19:54:30.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:30.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:30.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:30.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:30.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:30.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:30.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:30.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:30.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:30.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:30.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:30.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:30.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:30.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:30.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:30.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:30.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:30.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:30.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:30.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:30.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:30.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=39462], [object_number=37, tag_tid=2, entry_tid=39461], [object_number=36, tag_tid=2, entry_tid=39460], [object_number=39, tag_tid=2, entry_tid=39459]]], state=connected] 2026-02-22T19:54:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:30.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:30.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:30.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:30.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:30.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:30.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=39462], [object_number=37, tag_tid=2, entry_tid=39461], [object_number=36, tag_tid=2, entry_tid=39460], [object_number=39, tag_tid=2, entry_tid=39459]]], state=connected] 2026-02-22T19:54:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=2, entry_tid=39462' 2026-02-22T19:54:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=38, tag_tid=2, entry_tid=39462' 2026-02-22T19:54:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=39462' '!=' 'object_number=36, tag_tid=2, entry_tid=39196' 2026-02-22T19:54:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=39462' = 'object_number=36, tag_tid=2, entry_tid=39196' 2026-02-22T19:54:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=38, tag_tid=2, entry_tid=39462' 2026-02-22T19:54:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=39462' 2026-02-22T19:54:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:30.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=39462' 2026-02-22T19:54:30.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:30.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39462 2026-02-22T19:54:30.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 39462 2026-02-22T19:54:30.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:30.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:30.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:30.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:30.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:30.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:30.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:30.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:30.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:30.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:30.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:30.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:30.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:30.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:30.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:30.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:30.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:30.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:30.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:30.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:30.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:30.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:30.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:30.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:30.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:30.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:30.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=2, entry_tid=39732], [object_number=39, tag_tid=2, entry_tid=39731], [object_number=38, tag_tid=2, entry_tid=39730], [object_number=37, tag_tid=2, entry_tid=39729]]], state=connected] 2026-02-22T19:54:30.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:30.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:30.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:30.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:30.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:30.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:30.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:30.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:30.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=2, entry_tid=39732], [object_number=39, tag_tid=2, entry_tid=39731], [object_number=38, tag_tid=2, entry_tid=39730], [object_number=37, tag_tid=2, entry_tid=39729]]], state=connected] 2026-02-22T19:54:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=36, tag_tid=2, entry_tid=39732' 2026-02-22T19:54:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=36, tag_tid=2, entry_tid=39732' 2026-02-22T19:54:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=2, entry_tid=39732' '!=' 'object_number=38, tag_tid=2, entry_tid=39462' 2026-02-22T19:54:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:30.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=2, entry_tid=39732' = 'object_number=38, tag_tid=2, entry_tid=39462' 2026-02-22T19:54:30.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=36, tag_tid=2, entry_tid=39732' 2026-02-22T19:54:30.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:30.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:30.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=2, entry_tid=39732' 2026-02-22T19:54:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:30.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:30.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:30.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:30.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=2, entry_tid=39732' 2026-02-22T19:54:30.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:30.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:30.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39732 2026-02-22T19:54:30.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 39732 2026-02-22T19:54:30.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:30.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:30.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:31.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:31.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:31.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:31.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:31.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:31.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:31.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:31.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=39998], [object_number=37, tag_tid=2, entry_tid=39997], [object_number=36, tag_tid=2, entry_tid=39996], [object_number=39, tag_tid=2, entry_tid=39995]]], state=connected] 2026-02-22T19:54:31.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:31.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:31.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:31.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:31.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:31.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:31.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:31.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:31.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:31.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=39998], [object_number=37, tag_tid=2, entry_tid=39997], [object_number=36, tag_tid=2, entry_tid=39996], [object_number=39, tag_tid=2, entry_tid=39995]]], state=connected] 2026-02-22T19:54:31.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=2, entry_tid=39998' 2026-02-22T19:54:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=38, tag_tid=2, entry_tid=39998' 2026-02-22T19:54:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=39998' '!=' 'object_number=36, tag_tid=2, entry_tid=39732' 2026-02-22T19:54:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=39998' = 'object_number=36, tag_tid=2, entry_tid=39732' 2026-02-22T19:54:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=38, tag_tid=2, entry_tid=39998' 2026-02-22T19:54:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=39998' 2026-02-22T19:54:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:31.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:31.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:31.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:31.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:31.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:31.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:31.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=39998' 2026-02-22T19:54:31.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:31.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:31.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39998 2026-02-22T19:54:31.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 39998 2026-02-22T19:54:31.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:31.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:31.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:31.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:31.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:31.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:31.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:31.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:31.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:31.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:31.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:31.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:31.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:31.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:31.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=2, entry_tid=40244], [object_number=39, tag_tid=2, entry_tid=40243], [object_number=38, tag_tid=2, entry_tid=40242], [object_number=37, tag_tid=2, entry_tid=40241]]], state=connected] 2026-02-22T19:54:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:31.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:31.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:31.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:31.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:31.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:31.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=2, entry_tid=40244], [object_number=39, tag_tid=2, entry_tid=40243], [object_number=38, tag_tid=2, entry_tid=40242], [object_number=37, tag_tid=2, entry_tid=40241]]], state=connected] 2026-02-22T19:54:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=36, tag_tid=2, entry_tid=40244' 2026-02-22T19:54:31.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=36, tag_tid=2, entry_tid=40244' 2026-02-22T19:54:31.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=2, entry_tid=40244' '!=' 'object_number=38, tag_tid=2, entry_tid=39998' 2026-02-22T19:54:31.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:31.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=2, entry_tid=40244' = 'object_number=38, tag_tid=2, entry_tid=39998' 2026-02-22T19:54:31.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=36, tag_tid=2, entry_tid=40244' 2026-02-22T19:54:31.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:31.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:31.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:31.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:31.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=2, entry_tid=40244' 2026-02-22T19:54:31.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:31.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:31.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:31.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:31.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:31.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:31.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=2, entry_tid=40244' 2026-02-22T19:54:31.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:31.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:31.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40244 2026-02-22T19:54:31.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 40244 2026-02-22T19:54:31.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:31.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:31.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:31.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:31.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:31.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:31.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:31.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:31.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:31.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:31.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:31.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:31.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:31.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:31.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:31.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:31.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:31.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:31.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:31.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:31.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:31.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=40502], [object_number=37, tag_tid=2, entry_tid=40501], [object_number=36, tag_tid=2, entry_tid=40500], [object_number=39, tag_tid=2, entry_tid=40499]]], state=connected] 2026-02-22T19:54:31.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:31.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:31.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:31.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:31.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:31.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:31.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:31.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=40502], [object_number=37, tag_tid=2, entry_tid=40501], [object_number=36, tag_tid=2, entry_tid=40500], [object_number=39, tag_tid=2, entry_tid=40499]]], state=connected] 2026-02-22T19:54:32.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=2, entry_tid=40502' 2026-02-22T19:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=38, tag_tid=2, entry_tid=40502' 2026-02-22T19:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=40502' '!=' 'object_number=36, tag_tid=2, entry_tid=40244' 2026-02-22T19:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=40502' = 'object_number=36, tag_tid=2, entry_tid=40244' 2026-02-22T19:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=38, tag_tid=2, entry_tid=40502' 2026-02-22T19:54:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:32.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=40502' 2026-02-22T19:54:32.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:32.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:32.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:32.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:32.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=40502' 2026-02-22T19:54:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40502 2026-02-22T19:54:32.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 40502 2026-02-22T19:54:32.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:32.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:32.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:32.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:32.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:32.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:32.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:32.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:32.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:32.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:32.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:32.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:32.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:32.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:32.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:32.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:32.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:32.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:32.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=40763], [object_number=38, tag_tid=2, entry_tid=40762], [object_number=37, tag_tid=2, entry_tid=40761], [object_number=36, tag_tid=2, entry_tid=40760]]], state=connected] 2026-02-22T19:54:32.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:32.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:32.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:32.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:32.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:32.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:32.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:32.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:32.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:32.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:32.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:32.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:32.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:32.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:32.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:32.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:32.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:32.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:32.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=40763], [object_number=38, tag_tid=2, entry_tid=40762], [object_number=37, tag_tid=2, entry_tid=40761], [object_number=36, tag_tid=2, entry_tid=40760]]], state=connected] 2026-02-22T19:54:32.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=39, tag_tid=2, entry_tid=40763' 2026-02-22T19:54:32.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=39, tag_tid=2, entry_tid=40763' 2026-02-22T19:54:32.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=40763' '!=' 'object_number=38, tag_tid=2, entry_tid=40502' 2026-02-22T19:54:32.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:32.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=40763' = 'object_number=38, tag_tid=2, entry_tid=40502' 2026-02-22T19:54:32.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=39, tag_tid=2, entry_tid=40763' 2026-02-22T19:54:32.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:32.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:32.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:32.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=40763' 2026-02-22T19:54:32.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:32.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:32.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:32.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:32.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:32.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=40763' 2026-02-22T19:54:32.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:32.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40763 2026-02-22T19:54:32.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 40763 2026-02-22T19:54:32.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:32.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:32.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:32.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:32.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:32.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:32.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:32.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:32.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:32.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:32.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:32.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:32.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:32.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:32.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:32.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:32.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:32.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:32.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:32.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:32.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:32.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:32.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:32.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:32.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:32.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:32.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:32.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:32.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:32.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:32.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:32.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:32.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:32.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:32.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:32.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:32.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:32.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:32.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:32.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:32.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:32.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=40979], [object_number=38, tag_tid=2, entry_tid=40978], [object_number=37, tag_tid=2, entry_tid=40977], [object_number=36, tag_tid=2, entry_tid=40976]]], state=connected] 2026-02-22T19:54:32.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:32.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:32.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:32.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:32.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:32.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:32.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:32.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:32.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:32.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:32.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:32.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:32.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:32.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:32.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:32.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:32.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:32.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:32.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=2, entry_tid=40979], [object_number=38, tag_tid=2, entry_tid=40978], [object_number=37, tag_tid=2, entry_tid=40977], [object_number=36, tag_tid=2, entry_tid=40976]]], state=connected] 2026-02-22T19:54:32.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=39, tag_tid=2, entry_tid=40979' 2026-02-22T19:54:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=39, tag_tid=2, entry_tid=40979' 2026-02-22T19:54:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=40979' '!=' 'object_number=39, tag_tid=2, entry_tid=40763' 2026-02-22T19:54:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=2, entry_tid=40979' = 'object_number=39, tag_tid=2, entry_tid=40763' 2026-02-22T19:54:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=39, tag_tid=2, entry_tid=40979' 2026-02-22T19:54:32.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:32.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:32.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:32.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=40979' 2026-02-22T19:54:32.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:32.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:32.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:32.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:32.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=2, entry_tid=40979' 2026-02-22T19:54:32.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:32.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:32.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40979 2026-02-22T19:54:32.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 40979 2026-02-22T19:54:32.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:32.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:32.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:32.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:32.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:32.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:32.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:32.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:32.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:33.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:33.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:33.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:33.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:33.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:33.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:33.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=41222], [object_number=37, tag_tid=2, entry_tid=41221], [object_number=36, tag_tid=2, entry_tid=41220], [object_number=39, tag_tid=2, entry_tid=41219]]], state=connected] 2026-02-22T19:54:33.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:33.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:33.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=2, entry_tid=41222], [object_number=37, tag_tid=2, entry_tid=41221], [object_number=36, tag_tid=2, entry_tid=41220], [object_number=39, tag_tid=2, entry_tid=41219]]], state=connected] 2026-02-22T19:54:33.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=2, entry_tid=41222' 2026-02-22T19:54:33.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=38, tag_tid=2, entry_tid=41222' 2026-02-22T19:54:33.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=41222' '!=' 'object_number=39, tag_tid=2, entry_tid=40979' 2026-02-22T19:54:33.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:33.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=2, entry_tid=41222' = 'object_number=39, tag_tid=2, entry_tid=40979' 2026-02-22T19:54:33.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=38, tag_tid=2, entry_tid=41222' 2026-02-22T19:54:33.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:33.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:33.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:33.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:33.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=41222' 2026-02-22T19:54:33.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:33.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:33.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:33.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:33.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:33.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:33.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:33.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=2, entry_tid=41222' 2026-02-22T19:54:33.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:33.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41222 2026-02-22T19:54:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 41222 2026-02-22T19:54:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:33.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:33.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:33.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:33.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:33.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:33.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:33.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:33.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:33.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:33.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:33.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:33.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:33.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:33.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:33.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:33.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=41465], [object_number=36, tag_tid=2, entry_tid=41464], [object_number=39, tag_tid=2, entry_tid=41463], [object_number=38, tag_tid=2, entry_tid=41462]]], state=connected] 2026-02-22T19:54:33.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:33.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:33.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:33.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:33.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:33.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:33.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:33.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:33.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T19:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=41465], [object_number=36, tag_tid=2, entry_tid=41464], [object_number=39, tag_tid=2, entry_tid=41463], [object_number=38, tag_tid=2, entry_tid=41462]]], state=connected] 2026-02-22T19:54:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=37, tag_tid=2, entry_tid=41465' 2026-02-22T19:54:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=37, tag_tid=2, entry_tid=41465' 2026-02-22T19:54:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=37, tag_tid=2, entry_tid=41465' '!=' 'object_number=38, tag_tid=2, entry_tid=41222' 2026-02-22T19:54:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=37, tag_tid=2, entry_tid=41465' = 'object_number=38, tag_tid=2, entry_tid=41222' 2026-02-22T19:54:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=37, tag_tid=2, entry_tid=41465' 2026-02-22T19:54:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:33.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:33.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=41465' 2026-02-22T19:54:33.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:33.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:33.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:33.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:33.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:33.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:33.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:33.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=41465' 2026-02-22T19:54:33.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:33.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41465 2026-02-22T19:54:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 41465 2026-02-22T19:54:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:33.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:33.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:33.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:33.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:33.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:33.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:33.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:33.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:33.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:33.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:33.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:33.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:33.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:33.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:33.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:33.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:33.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:33.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:33.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:33.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:33.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:33.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:33.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:33.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:33.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:33.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:33.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:33.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:33.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:33.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:33.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:33.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:33.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:33.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:33.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:33.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:33.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:33.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:33.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:33.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:33.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:33.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:33.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:33.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:33.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:33.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:33.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:33.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:33.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:33.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:33.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:33.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:33.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:34.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:34.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:34.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:34.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=2, entry_tid=41746], [object_number=41, tag_tid=2, entry_tid=41745], [object_number=40, tag_tid=2, entry_tid=41744], [object_number=43, tag_tid=2, entry_tid=41743]]], state=connected] 2026-02-22T19:54:34.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:34.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:34.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:34.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:34.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=2, entry_tid=41746], [object_number=41, tag_tid=2, entry_tid=41745], [object_number=40, tag_tid=2, entry_tid=41744], [object_number=43, tag_tid=2, entry_tid=41743]]], state=connected] 2026-02-22T19:54:34.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=42, tag_tid=2, entry_tid=41746' 2026-02-22T19:54:34.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=42, tag_tid=2, entry_tid=41746' 2026-02-22T19:54:34.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=2, entry_tid=41746' '!=' 'object_number=37, tag_tid=2, entry_tid=41465' 2026-02-22T19:54:34.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:34.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=2, entry_tid=41746' = 'object_number=37, tag_tid=2, entry_tid=41465' 2026-02-22T19:54:34.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=42, tag_tid=2, entry_tid=41746' 2026-02-22T19:54:34.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:34.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:34.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:34.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:34.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=42, tag_tid=2, entry_tid=41746' 2026-02-22T19:54:34.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:34.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:34.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:34.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:34.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:34.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:34.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=42, tag_tid=2, entry_tid=41746' 2026-02-22T19:54:34.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:34.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41746 2026-02-22T19:54:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 41746 2026-02-22T19:54:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:34.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:34.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:34.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:34.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:34.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:34.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:34.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:34.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:34.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:34.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:34.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:34.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:34.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:34.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:34.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:34.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:34.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:34.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:34.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=40, tag_tid=2, entry_tid=42252], [object_number=43, tag_tid=2, entry_tid=42251], [object_number=42, tag_tid=2, entry_tid=42250], [object_number=41, tag_tid=2, entry_tid=42249]]], state=connected] 2026-02-22T19:54:34.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:34.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:34.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:34.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:34.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:34.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:34.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:34.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:34.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:34.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:34.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:34.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:34.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:34.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:34.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:34.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:34.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:34.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:34.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=40, tag_tid=2, entry_tid=42252], [object_number=43, tag_tid=2, entry_tid=42251], [object_number=42, tag_tid=2, entry_tid=42250], [object_number=41, tag_tid=2, entry_tid=42249]]], state=connected] 2026-02-22T19:54:34.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:34.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=40, tag_tid=2, entry_tid=42252' 2026-02-22T19:54:34.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=40, tag_tid=2, entry_tid=42252' 2026-02-22T19:54:34.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=40, tag_tid=2, entry_tid=42252' '!=' 'object_number=42, tag_tid=2, entry_tid=41746' 2026-02-22T19:54:34.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:34.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=40, tag_tid=2, entry_tid=42252' = 'object_number=42, tag_tid=2, entry_tid=41746' 2026-02-22T19:54:34.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=40, tag_tid=2, entry_tid=42252' 2026-02-22T19:54:34.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:34.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:34.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:34.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:34.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=40, tag_tid=2, entry_tid=42252' 2026-02-22T19:54:34.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:34.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:34.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:34.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:34.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:34.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:34.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:34.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=40, tag_tid=2, entry_tid=42252' 2026-02-22T19:54:34.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:34.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:34.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42252 2026-02-22T19:54:34.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 42252 2026-02-22T19:54:34.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:34.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:34.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:34.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:34.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:34.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:34.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:34.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:35.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:35.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:35.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:35.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:35.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:35.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:35.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:35.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:35.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:35.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:35.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:35.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=42513], [object_number=40, tag_tid=2, entry_tid=42512], [object_number=43, tag_tid=2, entry_tid=42511], [object_number=42, tag_tid=2, entry_tid=42510]]], state=connected] 2026-02-22T19:54:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:35.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:35.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:35.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:35.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:35.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:35.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:35.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:35.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:35.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:35.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=42513], [object_number=40, tag_tid=2, entry_tid=42512], [object_number=43, tag_tid=2, entry_tid=42511], [object_number=42, tag_tid=2, entry_tid=42510]]], state=connected] 2026-02-22T19:54:35.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=42513' 2026-02-22T19:54:35.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=42513' 2026-02-22T19:54:35.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=42513' '!=' 'object_number=40, tag_tid=2, entry_tid=42252' 2026-02-22T19:54:35.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:35.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=42513' = 'object_number=40, tag_tid=2, entry_tid=42252' 2026-02-22T19:54:35.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=41, tag_tid=2, entry_tid=42513' 2026-02-22T19:54:35.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:35.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:35.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:35.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=42513' 2026-02-22T19:54:35.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:35.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:35.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:35.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=42513' 2026-02-22T19:54:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42513 2026-02-22T19:54:35.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 42513 2026-02-22T19:54:35.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:35.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:35.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:35.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:35.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:35.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:35.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:35.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:35.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:35.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:35.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=42749], [object_number=40, tag_tid=2, entry_tid=42748], [object_number=43, tag_tid=2, entry_tid=42747], [object_number=42, tag_tid=2, entry_tid=42746]]], state=connected] 2026-02-22T19:54:35.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:35.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:35.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:35.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=42749], [object_number=40, tag_tid=2, entry_tid=42748], [object_number=43, tag_tid=2, entry_tid=42747], [object_number=42, tag_tid=2, entry_tid=42746]]], state=connected] 2026-02-22T19:54:35.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=42749' 2026-02-22T19:54:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=42749' 2026-02-22T19:54:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=42749' '!=' 'object_number=41, tag_tid=2, entry_tid=42513' 2026-02-22T19:54:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=42749' = 'object_number=41, tag_tid=2, entry_tid=42513' 2026-02-22T19:54:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=41, tag_tid=2, entry_tid=42749' 2026-02-22T19:54:35.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:35.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:35.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:35.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=42749' 2026-02-22T19:54:35.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:35.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:35.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:35.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:35.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=42749' 2026-02-22T19:54:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42749 2026-02-22T19:54:35.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 42749 2026-02-22T19:54:35.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:35.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:35.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:35.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:35.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:35.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:35.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:35.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:35.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:35.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:35.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:35.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:35.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:35.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:35.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:35.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:35.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:35.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=42997], [object_number=40, tag_tid=2, entry_tid=42996], [object_number=43, tag_tid=2, entry_tid=42995], [object_number=42, tag_tid=2, entry_tid=42994]]], state=connected] 2026-02-22T19:54:35.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:35.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:35.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:35.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:35.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:35.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:35.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:35.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:35.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:35.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:35.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:35.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=42997], [object_number=40, tag_tid=2, entry_tid=42996], [object_number=43, tag_tid=2, entry_tid=42995], [object_number=42, tag_tid=2, entry_tid=42994]]], state=connected] 2026-02-22T19:54:35.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:35.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=42997' 2026-02-22T19:54:35.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=42997' 2026-02-22T19:54:35.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=42997' '!=' 'object_number=41, tag_tid=2, entry_tid=42749' 2026-02-22T19:54:35.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:35.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=42997' = 'object_number=41, tag_tid=2, entry_tid=42749' 2026-02-22T19:54:35.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=41, tag_tid=2, entry_tid=42997' 2026-02-22T19:54:35.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:35.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:35.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:35.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=42997' 2026-02-22T19:54:35.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:35.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:35.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:35.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:35.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:35.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=42997' 2026-02-22T19:54:35.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:35.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:35.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42997 2026-02-22T19:54:35.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 42997 2026-02-22T19:54:35.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:35.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:35.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:36.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:36.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:36.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:36.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:36.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:36.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:36.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:36.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:36.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:36.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:36.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:36.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:36.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:36.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:36.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:36.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:36.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:36.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:36.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=43241], [object_number=40, tag_tid=2, entry_tid=43240], [object_number=43, tag_tid=2, entry_tid=43239], [object_number=42, tag_tid=2, entry_tid=43238]]], state=connected] 2026-02-22T19:54:36.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:36.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:36.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:36.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:36.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:36.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:36.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=43241], [object_number=40, tag_tid=2, entry_tid=43240], [object_number=43, tag_tid=2, entry_tid=43239], [object_number=42, tag_tid=2, entry_tid=43238]]], state=connected] 2026-02-22T19:54:36.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=43241' 2026-02-22T19:54:36.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=43241' 2026-02-22T19:54:36.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=43241' '!=' 'object_number=41, tag_tid=2, entry_tid=42997' 2026-02-22T19:54:36.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:36.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=43241' = 'object_number=41, tag_tid=2, entry_tid=42997' 2026-02-22T19:54:36.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=41, tag_tid=2, entry_tid=43241' 2026-02-22T19:54:36.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:36.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:36.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:36.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=43241' 2026-02-22T19:54:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:36.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:36.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:36.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:36.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:36.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=43241' 2026-02-22T19:54:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43241 2026-02-22T19:54:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 43241 2026-02-22T19:54:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:36.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:36.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:36.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:36.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:36.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:36.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:36.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:36.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:36.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:36.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:36.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:36.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:36.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:36.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:36.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:36.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:36.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:36.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:36.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:36.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:36.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:36.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:36.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:36.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:36.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:36.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:36.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:36.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:36.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:36.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:36.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:36.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:36.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:36.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:36.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:36.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:36.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:36.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:36.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:36.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:36.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:36.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:36.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=43661], [object_number=40, tag_tid=2, entry_tid=43660], [object_number=43, tag_tid=2, entry_tid=43659], [object_number=42, tag_tid=2, entry_tid=43658]]], state=connected] 2026-02-22T19:54:36.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:36.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:36.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:36.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:36.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:36.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:36.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:36.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:36.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:36.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:36.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:36.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:36.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:36.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:36.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:36.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:36.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:36.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:36.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=43661], [object_number=40, tag_tid=2, entry_tid=43660], [object_number=43, tag_tid=2, entry_tid=43659], [object_number=42, tag_tid=2, entry_tid=43658]]], state=connected] 2026-02-22T19:54:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:36.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=43661' 2026-02-22T19:54:36.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=43661' 2026-02-22T19:54:36.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=43661' '!=' 'object_number=41, tag_tid=2, entry_tid=43241' 2026-02-22T19:54:36.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:36.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=43661' = 'object_number=41, tag_tid=2, entry_tid=43241' 2026-02-22T19:54:36.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=41, tag_tid=2, entry_tid=43661' 2026-02-22T19:54:36.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:36.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:36.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:36.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:36.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=43661' 2026-02-22T19:54:36.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:36.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:37.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:37.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:37.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=43661' 2026-02-22T19:54:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43661 2026-02-22T19:54:37.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 43661 2026-02-22T19:54:37.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:37.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:37.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:37.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:37.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:37.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:37.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:37.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:37.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:37.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:37.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:37.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:37.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:37.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:37.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=44133], [object_number=40, tag_tid=2, entry_tid=44132], [object_number=43, tag_tid=2, entry_tid=44131], [object_number=42, tag_tid=2, entry_tid=44130]]], state=connected] 2026-02-22T19:54:37.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:37.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:37.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:37.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:37.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:37.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=44133], [object_number=40, tag_tid=2, entry_tid=44132], [object_number=43, tag_tid=2, entry_tid=44131], [object_number=42, tag_tid=2, entry_tid=44130]]], state=connected] 2026-02-22T19:54:37.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=44133' 2026-02-22T19:54:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=44133' 2026-02-22T19:54:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=44133' '!=' 'object_number=41, tag_tid=2, entry_tid=43661' 2026-02-22T19:54:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=44133' = 'object_number=41, tag_tid=2, entry_tid=43661' 2026-02-22T19:54:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=41, tag_tid=2, entry_tid=44133' 2026-02-22T19:54:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:37.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=44133' 2026-02-22T19:54:37.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:37.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=44133' 2026-02-22T19:54:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44133 2026-02-22T19:54:37.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 44133 2026-02-22T19:54:37.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:37.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:37.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:37.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:37.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:37.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:37.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:37.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:37.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:37.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:37.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=43, tag_tid=2, entry_tid=44439], [object_number=42, tag_tid=2, entry_tid=44438], [object_number=41, tag_tid=2, entry_tid=44437], [object_number=40, tag_tid=2, entry_tid=44436]]], state=connected] 2026-02-22T19:54:37.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:37.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:37.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:37.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:37.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:37.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:37.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:37.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:37.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:37.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:37.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:37.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=43, tag_tid=2, entry_tid=44439], [object_number=42, tag_tid=2, entry_tid=44438], [object_number=41, tag_tid=2, entry_tid=44437], [object_number=40, tag_tid=2, entry_tid=44436]]], state=connected] 2026-02-22T19:54:37.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:37.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=43, tag_tid=2, entry_tid=44439' 2026-02-22T19:54:37.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=43, tag_tid=2, entry_tid=44439' 2026-02-22T19:54:37.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=43, tag_tid=2, entry_tid=44439' '!=' 'object_number=41, tag_tid=2, entry_tid=44133' 2026-02-22T19:54:37.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:37.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=43, tag_tid=2, entry_tid=44439' = 'object_number=41, tag_tid=2, entry_tid=44133' 2026-02-22T19:54:37.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=43, tag_tid=2, entry_tid=44439' 2026-02-22T19:54:37.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:37.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:37.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:37.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=43, tag_tid=2, entry_tid=44439' 2026-02-22T19:54:37.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:37.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:37.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:37.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:37.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:37.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=43, tag_tid=2, entry_tid=44439' 2026-02-22T19:54:37.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:37.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:37.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44439 2026-02-22T19:54:37.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 44439 2026-02-22T19:54:37.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:37.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:37.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:37.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:37.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:37.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:37.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:37.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:37.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:37.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:37.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:37.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:38.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=43, tag_tid=2, entry_tid=44727], [object_number=42, tag_tid=2, entry_tid=44726], [object_number=41, tag_tid=2, entry_tid=44725], [object_number=40, tag_tid=2, entry_tid=44724]]], state=connected] 2026-02-22T19:54:38.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:38.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:38.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:38.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:38.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:38.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:38.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:38.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:38.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:38.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:38.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:38.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:38.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=43, tag_tid=2, entry_tid=44727], [object_number=42, tag_tid=2, entry_tid=44726], [object_number=41, tag_tid=2, entry_tid=44725], [object_number=40, tag_tid=2, entry_tid=44724]]], state=connected] 2026-02-22T19:54:38.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=43, tag_tid=2, entry_tid=44727' 2026-02-22T19:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=43, tag_tid=2, entry_tid=44727' 2026-02-22T19:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=43, tag_tid=2, entry_tid=44727' '!=' 'object_number=43, tag_tid=2, entry_tid=44439' 2026-02-22T19:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=43, tag_tid=2, entry_tid=44727' = 'object_number=43, tag_tid=2, entry_tid=44439' 2026-02-22T19:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=43, tag_tid=2, entry_tid=44727' 2026-02-22T19:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=43, tag_tid=2, entry_tid=44727' 2026-02-22T19:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:38.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:38.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:38.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:38.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=43, tag_tid=2, entry_tid=44727' 2026-02-22T19:54:38.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:38.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44727 2026-02-22T19:54:38.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 44727 2026-02-22T19:54:38.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:38.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:38.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:38.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:38.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:38.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:38.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:38.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:38.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:38.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:38.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:38.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:38.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:38.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:38.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:38.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:38.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=44977], [object_number=40, tag_tid=2, entry_tid=44976], [object_number=43, tag_tid=2, entry_tid=44975], [object_number=42, tag_tid=2, entry_tid=44974]]], state=connected] 2026-02-22T19:54:38.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:38.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:38.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:38.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:38.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:38.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:38.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:38.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:38.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:38.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:38.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:38.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:38.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=44977], [object_number=40, tag_tid=2, entry_tid=44976], [object_number=43, tag_tid=2, entry_tid=44975], [object_number=42, tag_tid=2, entry_tid=44974]]], state=connected] 2026-02-22T19:54:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=44977' 2026-02-22T19:54:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=44977' 2026-02-22T19:54:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=44977' '!=' 'object_number=43, tag_tid=2, entry_tid=44727' 2026-02-22T19:54:38.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:38.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=44977' = 'object_number=43, tag_tid=2, entry_tid=44727' 2026-02-22T19:54:38.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=41, tag_tid=2, entry_tid=44977' 2026-02-22T19:54:38.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:38.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:38.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=44977' 2026-02-22T19:54:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:38.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:38.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:38.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:38.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=44977' 2026-02-22T19:54:38.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:38.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44977 2026-02-22T19:54:38.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 44977 2026-02-22T19:54:38.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:38.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:38.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:38.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:38.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:38.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:38.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:38.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:38.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:38.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:38.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:38.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:38.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:38.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:38.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:38.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:38.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:38.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=43, tag_tid=2, entry_tid=45199], [object_number=42, tag_tid=2, entry_tid=45198], [object_number=41, tag_tid=2, entry_tid=45197], [object_number=40, tag_tid=2, entry_tid=45196]]], state=connected] 2026-02-22T19:54:38.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=43, tag_tid=2, entry_tid=45199], [object_number=42, tag_tid=2, entry_tid=45198], [object_number=41, tag_tid=2, entry_tid=45197], [object_number=40, tag_tid=2, entry_tid=45196]]], state=connected] 2026-02-22T19:54:38.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=43, tag_tid=2, entry_tid=45199' 2026-02-22T19:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=43, tag_tid=2, entry_tid=45199' 2026-02-22T19:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=43, tag_tid=2, entry_tid=45199' '!=' 'object_number=41, tag_tid=2, entry_tid=44977' 2026-02-22T19:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=43, tag_tid=2, entry_tid=45199' = 'object_number=41, tag_tid=2, entry_tid=44977' 2026-02-22T19:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=43, tag_tid=2, entry_tid=45199' 2026-02-22T19:54:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:38.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=43, tag_tid=2, entry_tid=45199' 2026-02-22T19:54:38.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:38.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:38.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:38.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:38.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:38.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=43, tag_tid=2, entry_tid=45199' 2026-02-22T19:54:38.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:38.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45199 2026-02-22T19:54:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 45199 2026-02-22T19:54:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:39.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:39.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:39.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:39.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:39.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:39.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:39.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:39.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:39.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:39.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:39.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:39.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:39.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=45425], [object_number=40, tag_tid=2, entry_tid=45424], [object_number=43, tag_tid=2, entry_tid=45423], [object_number=42, tag_tid=2, entry_tid=45422]]], state=connected] 2026-02-22T19:54:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:39.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:39.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:39.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:39.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=45425], [object_number=40, tag_tid=2, entry_tid=45424], [object_number=43, tag_tid=2, entry_tid=45423], [object_number=42, tag_tid=2, entry_tid=45422]]], state=connected] 2026-02-22T19:54:39.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=45425' 2026-02-22T19:54:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=45425' 2026-02-22T19:54:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=45425' '!=' 'object_number=43, tag_tid=2, entry_tid=45199' 2026-02-22T19:54:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=2, entry_tid=45425' = 'object_number=43, tag_tid=2, entry_tid=45199' 2026-02-22T19:54:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=41, tag_tid=2, entry_tid=45425' 2026-02-22T19:54:39.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:39.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:39.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:39.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=45425' 2026-02-22T19:54:39.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:39.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:39.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:39.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:39.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=45425' 2026-02-22T19:54:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45425 2026-02-22T19:54:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 45425 2026-02-22T19:54:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:39.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:39.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:39.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:39.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:39.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:39.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:39.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:39.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:39.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:39.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:39.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:39.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:39.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:39.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:39.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=2, entry_tid=45678], [object_number=41, tag_tid=2, entry_tid=45677], [object_number=40, tag_tid=2, entry_tid=45676], [object_number=43, tag_tid=2, entry_tid=45675]]], state=connected] 2026-02-22T19:54:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:39.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:39.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:39.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:39.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:39.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:39.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:39.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:39.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=2, entry_tid=45678], [object_number=41, tag_tid=2, entry_tid=45677], [object_number=40, tag_tid=2, entry_tid=45676], [object_number=43, tag_tid=2, entry_tid=45675]]], state=connected] 2026-02-22T19:54:39.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=42, tag_tid=2, entry_tid=45678' 2026-02-22T19:54:39.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=42, tag_tid=2, entry_tid=45678' 2026-02-22T19:54:39.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=2, entry_tid=45678' '!=' 'object_number=41, tag_tid=2, entry_tid=45425' 2026-02-22T19:54:39.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:39.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=2, entry_tid=45678' = 'object_number=41, tag_tid=2, entry_tid=45425' 2026-02-22T19:54:39.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=42, tag_tid=2, entry_tid=45678' 2026-02-22T19:54:39.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:39.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:39.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:39.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=42, tag_tid=2, entry_tid=45678' 2026-02-22T19:54:39.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:39.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=42, tag_tid=2, entry_tid=45678' 2026-02-22T19:54:39.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:39.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45678 2026-02-22T19:54:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 45678 2026-02-22T19:54:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:39.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:39.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:39.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:39.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:39.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:39.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:39.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:39.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:39.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:39.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:39.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:39.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:39.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:39.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:39.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=47, tag_tid=2, entry_tid=45815], [object_number=46, tag_tid=2, entry_tid=45814], [object_number=45, tag_tid=2, entry_tid=45813], [object_number=40, tag_tid=2, entry_tid=45812]]], state=connected] 2026-02-22T19:54:39.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:39.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:39.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:39.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:39.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:39.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:39.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:39.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:39.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:39.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:39.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:39.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T19:54:39.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:39.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:39.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:39.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=47, tag_tid=2, entry_tid=45815], [object_number=46, tag_tid=2, entry_tid=45814], [object_number=45, tag_tid=2, entry_tid=45813], [object_number=40, tag_tid=2, entry_tid=45812]]], state=connected] 2026-02-22T19:54:39.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=47, tag_tid=2, entry_tid=45815' 2026-02-22T19:54:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=47, tag_tid=2, entry_tid=45815' 2026-02-22T19:54:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=47, tag_tid=2, entry_tid=45815' '!=' 'object_number=42, tag_tid=2, entry_tid=45678' 2026-02-22T19:54:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=47, tag_tid=2, entry_tid=45815' = 'object_number=42, tag_tid=2, entry_tid=45678' 2026-02-22T19:54:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=47, tag_tid=2, entry_tid=45815' 2026-02-22T19:54:39.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:39.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:39.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:39.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=47, tag_tid=2, entry_tid=45815' 2026-02-22T19:54:39.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:39.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:39.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:39.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:39.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:39.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=47, tag_tid=2, entry_tid=45815' 2026-02-22T19:54:39.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:39.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:39.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45815 2026-02-22T19:54:39.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 45815 2026-02-22T19:54:39.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:39.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:39.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:40.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:40.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:40.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:40.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:40.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:40.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:40.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:40.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:40.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:40.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:40.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:40.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:40.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:40.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:40.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:40.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:40.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:40.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=46053], [object_number=44, tag_tid=2, entry_tid=46052], [object_number=47, tag_tid=2, entry_tid=46051], [object_number=46, tag_tid=2, entry_tid=46050]]], state=connected] 2026-02-22T19:54:40.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:40.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:40.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:40.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:40.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:40.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:40.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=46053], [object_number=44, tag_tid=2, entry_tid=46052], [object_number=47, tag_tid=2, entry_tid=46051], [object_number=46, tag_tid=2, entry_tid=46050]]], state=connected] 2026-02-22T19:54:40.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=45, tag_tid=2, entry_tid=46053' 2026-02-22T19:54:40.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=45, tag_tid=2, entry_tid=46053' 2026-02-22T19:54:40.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=46053' '!=' 'object_number=47, tag_tid=2, entry_tid=45815' 2026-02-22T19:54:40.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:40.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=46053' = 'object_number=47, tag_tid=2, entry_tid=45815' 2026-02-22T19:54:40.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=45, tag_tid=2, entry_tid=46053' 2026-02-22T19:54:40.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:40.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:40.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:40.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=46053' 2026-02-22T19:54:40.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:40.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:40.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=46053' 2026-02-22T19:54:40.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:40.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46053 2026-02-22T19:54:40.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 46053 2026-02-22T19:54:40.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:40.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:40.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:40.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:40.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:40.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:40.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:40.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:40.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:40.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:40.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:40.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:40.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:40.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:40.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=44, tag_tid=2, entry_tid=46332], [object_number=47, tag_tid=2, entry_tid=46331], [object_number=46, tag_tid=2, entry_tid=46330], [object_number=45, tag_tid=2, entry_tid=46329]]], state=connected] 2026-02-22T19:54:40.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:40.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:40.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:40.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:40.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=44, tag_tid=2, entry_tid=46332], [object_number=47, tag_tid=2, entry_tid=46331], [object_number=46, tag_tid=2, entry_tid=46330], [object_number=45, tag_tid=2, entry_tid=46329]]], state=connected] 2026-02-22T19:54:40.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=44, tag_tid=2, entry_tid=46332' 2026-02-22T19:54:40.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=44, tag_tid=2, entry_tid=46332' 2026-02-22T19:54:40.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=44, tag_tid=2, entry_tid=46332' '!=' 'object_number=45, tag_tid=2, entry_tid=46053' 2026-02-22T19:54:40.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:40.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=44, tag_tid=2, entry_tid=46332' = 'object_number=45, tag_tid=2, entry_tid=46053' 2026-02-22T19:54:40.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=44, tag_tid=2, entry_tid=46332' 2026-02-22T19:54:40.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:40.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:40.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=44, tag_tid=2, entry_tid=46332' 2026-02-22T19:54:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:40.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=44, tag_tid=2, entry_tid=46332' 2026-02-22T19:54:40.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:40.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46332 2026-02-22T19:54:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 46332 2026-02-22T19:54:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:40.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:40.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:40.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:40.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:40.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:40.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:40.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:40.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:40.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:40.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:40.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:40.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:40.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:40.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:40.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:40.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:40.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=47, tag_tid=2, entry_tid=46595], [object_number=46, tag_tid=2, entry_tid=46594], [object_number=45, tag_tid=2, entry_tid=46593], [object_number=44, tag_tid=2, entry_tid=46592]]], state=connected] 2026-02-22T19:54:40.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:40.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=47, tag_tid=2, entry_tid=46595], [object_number=46, tag_tid=2, entry_tid=46594], [object_number=45, tag_tid=2, entry_tid=46593], [object_number=44, tag_tid=2, entry_tid=46592]]], state=connected] 2026-02-22T19:54:40.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=47, tag_tid=2, entry_tid=46595' 2026-02-22T19:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=47, tag_tid=2, entry_tid=46595' 2026-02-22T19:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=47, tag_tid=2, entry_tid=46595' '!=' 'object_number=44, tag_tid=2, entry_tid=46332' 2026-02-22T19:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=47, tag_tid=2, entry_tid=46595' = 'object_number=44, tag_tid=2, entry_tid=46332' 2026-02-22T19:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=47, tag_tid=2, entry_tid=46595' 2026-02-22T19:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=47, tag_tid=2, entry_tid=46595' 2026-02-22T19:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=47, tag_tid=2, entry_tid=46595' 2026-02-22T19:54:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:40.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46595 2026-02-22T19:54:40.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 46595 2026-02-22T19:54:40.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:40.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:40.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:41.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:41.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:41.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:41.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:41.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:41.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:41.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:41.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:41.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:41.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:41.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:41.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:41.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:41.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:41.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:41.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:41.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=44, tag_tid=2, entry_tid=46868], [object_number=47, tag_tid=2, entry_tid=46867], [object_number=46, tag_tid=2, entry_tid=46866], [object_number=45, tag_tid=2, entry_tid=46865]]], state=connected] 2026-02-22T19:54:41.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:41.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:41.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:41.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:41.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=44, tag_tid=2, entry_tid=46868], [object_number=47, tag_tid=2, entry_tid=46867], [object_number=46, tag_tid=2, entry_tid=46866], [object_number=45, tag_tid=2, entry_tid=46865]]], state=connected] 2026-02-22T19:54:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=44, tag_tid=2, entry_tid=46868' 2026-02-22T19:54:41.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=44, tag_tid=2, entry_tid=46868' 2026-02-22T19:54:41.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=44, tag_tid=2, entry_tid=46868' '!=' 'object_number=47, tag_tid=2, entry_tid=46595' 2026-02-22T19:54:41.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:41.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=44, tag_tid=2, entry_tid=46868' = 'object_number=47, tag_tid=2, entry_tid=46595' 2026-02-22T19:54:41.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=44, tag_tid=2, entry_tid=46868' 2026-02-22T19:54:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:41.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:41.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=44, tag_tid=2, entry_tid=46868' 2026-02-22T19:54:41.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:41.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:41.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:41.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:41.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:41.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=44, tag_tid=2, entry_tid=46868' 2026-02-22T19:54:41.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:41.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:41.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46868 2026-02-22T19:54:41.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 46868 2026-02-22T19:54:41.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:41.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:41.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:41.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:41.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:41.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:41.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:41.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:41.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:41.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:41.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:41.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:41.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:41.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:41.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:41.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:41.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:41.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:41.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:41.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:41.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:41.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:41.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:41.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:41.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:41.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:41.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:41.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:41.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:41.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:41.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:41.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:41.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=47122], [object_number=45, tag_tid=2, entry_tid=47121], [object_number=44, tag_tid=2, entry_tid=47120], [object_number=47, tag_tid=2, entry_tid=47119]]], state=connected] 2026-02-22T19:54:41.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:41.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:41.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:41.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:41.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:41.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:41.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:41.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:41.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=47122], [object_number=45, tag_tid=2, entry_tid=47121], [object_number=44, tag_tid=2, entry_tid=47120], [object_number=47, tag_tid=2, entry_tid=47119]]], state=connected] 2026-02-22T19:54:41.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=46, tag_tid=2, entry_tid=47122' 2026-02-22T19:54:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=46, tag_tid=2, entry_tid=47122' 2026-02-22T19:54:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=47122' '!=' 'object_number=44, tag_tid=2, entry_tid=46868' 2026-02-22T19:54:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=47122' = 'object_number=44, tag_tid=2, entry_tid=46868' 2026-02-22T19:54:41.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=46, tag_tid=2, entry_tid=47122' 2026-02-22T19:54:41.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:41.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:41.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:41.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:41.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=47122' 2026-02-22T19:54:41.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:41.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:41.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:41.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:41.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:41.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:41.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:41.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=47122' 2026-02-22T19:54:41.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:41.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:41.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47122 2026-02-22T19:54:41.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 47122 2026-02-22T19:54:41.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:41.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:41.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:41.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:41.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:41.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:41.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:41.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:41.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:41.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:41.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:41.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:41.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:41.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:41.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:41.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:41.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:41.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:41.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:41.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:41.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:41.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:41.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:41.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:41.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:41.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:42.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:42.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:42.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:42.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:42.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=47, tag_tid=2, entry_tid=47363], [object_number=46, tag_tid=2, entry_tid=47362], [object_number=45, tag_tid=2, entry_tid=47361], [object_number=44, tag_tid=2, entry_tid=47360]]], state=connected] 2026-02-22T19:54:42.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:42.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:42.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=47, tag_tid=2, entry_tid=47363], [object_number=46, tag_tid=2, entry_tid=47362], [object_number=45, tag_tid=2, entry_tid=47361], [object_number=44, tag_tid=2, entry_tid=47360]]], state=connected] 2026-02-22T19:54:42.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=47, tag_tid=2, entry_tid=47363' 2026-02-22T19:54:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=47, tag_tid=2, entry_tid=47363' 2026-02-22T19:54:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=47, tag_tid=2, entry_tid=47363' '!=' 'object_number=46, tag_tid=2, entry_tid=47122' 2026-02-22T19:54:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=47, tag_tid=2, entry_tid=47363' = 'object_number=46, tag_tid=2, entry_tid=47122' 2026-02-22T19:54:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=47, tag_tid=2, entry_tid=47363' 2026-02-22T19:54:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=47, tag_tid=2, entry_tid=47363' 2026-02-22T19:54:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:42.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:42.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:42.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:42.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:42.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=47, tag_tid=2, entry_tid=47363' 2026-02-22T19:54:42.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:42.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:42.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47363 2026-02-22T19:54:42.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 47363 2026-02-22T19:54:42.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:42.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:42.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:42.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:42.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:42.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:42.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:42.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:42.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:42.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:42.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:42.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:42.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:42.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:42.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:42.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:42.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:42.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:42.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:42.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=47, tag_tid=2, entry_tid=47627], [object_number=46, tag_tid=2, entry_tid=47626], [object_number=45, tag_tid=2, entry_tid=47625], [object_number=44, tag_tid=2, entry_tid=47624]]], state=connected] 2026-02-22T19:54:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:42.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:42.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:42.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:42.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:42.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:42.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:42.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:42.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=47, tag_tid=2, entry_tid=47627], [object_number=46, tag_tid=2, entry_tid=47626], [object_number=45, tag_tid=2, entry_tid=47625], [object_number=44, tag_tid=2, entry_tid=47624]]], state=connected] 2026-02-22T19:54:42.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=47, tag_tid=2, entry_tid=47627' 2026-02-22T19:54:42.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=47, tag_tid=2, entry_tid=47627' 2026-02-22T19:54:42.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=47, tag_tid=2, entry_tid=47627' '!=' 'object_number=47, tag_tid=2, entry_tid=47363' 2026-02-22T19:54:42.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:42.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=47, tag_tid=2, entry_tid=47627' = 'object_number=47, tag_tid=2, entry_tid=47363' 2026-02-22T19:54:42.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=47, tag_tid=2, entry_tid=47627' 2026-02-22T19:54:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=47, tag_tid=2, entry_tid=47627' 2026-02-22T19:54:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:42.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:42.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:42.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:42.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:42.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:42.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=47, tag_tid=2, entry_tid=47627' 2026-02-22T19:54:42.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:42.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:42.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47627 2026-02-22T19:54:42.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 47627 2026-02-22T19:54:42.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:42.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:42.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:42.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:42.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:42.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:42.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:42.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:42.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:42.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:42.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:42.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:42.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:42.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:42.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:42.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:42.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:42.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:42.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:42.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:42.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:42.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:42.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:42.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:42.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:42.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:42.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:42.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:43.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:43.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:43.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:43.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=47993], [object_number=44, tag_tid=2, entry_tid=47992], [object_number=47, tag_tid=2, entry_tid=47991], [object_number=46, tag_tid=2, entry_tid=47990]]], state=connected] 2026-02-22T19:54:43.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:43.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:43.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:43.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:43.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:43.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:43.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:43.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:43.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:43.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:43.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=47993], [object_number=44, tag_tid=2, entry_tid=47992], [object_number=47, tag_tid=2, entry_tid=47991], [object_number=46, tag_tid=2, entry_tid=47990]]], state=connected] 2026-02-22T19:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=45, tag_tid=2, entry_tid=47993' 2026-02-22T19:54:43.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=45, tag_tid=2, entry_tid=47993' 2026-02-22T19:54:43.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=47993' '!=' 'object_number=47, tag_tid=2, entry_tid=47627' 2026-02-22T19:54:43.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:43.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=47993' = 'object_number=47, tag_tid=2, entry_tid=47627' 2026-02-22T19:54:43.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=45, tag_tid=2, entry_tid=47993' 2026-02-22T19:54:43.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:43.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:43.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=47993' 2026-02-22T19:54:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:43.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:43.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:43.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:43.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=47993' 2026-02-22T19:54:43.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:43.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47993 2026-02-22T19:54:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 47993 2026-02-22T19:54:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:43.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:43.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:43.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:43.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:43.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:43.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:43.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:43.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:43.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:43.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:43.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:43.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:43.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:43.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:43.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=44, tag_tid=2, entry_tid=48244], [object_number=47, tag_tid=2, entry_tid=48243], [object_number=46, tag_tid=2, entry_tid=48242], [object_number=45, tag_tid=2, entry_tid=48241]]], state=connected] 2026-02-22T19:54:43.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:43.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:43.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:43.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:43.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:43.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=44, tag_tid=2, entry_tid=48244], [object_number=47, tag_tid=2, entry_tid=48243], [object_number=46, tag_tid=2, entry_tid=48242], [object_number=45, tag_tid=2, entry_tid=48241]]], state=connected] 2026-02-22T19:54:43.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=44, tag_tid=2, entry_tid=48244' 2026-02-22T19:54:43.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=44, tag_tid=2, entry_tid=48244' 2026-02-22T19:54:43.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=44, tag_tid=2, entry_tid=48244' '!=' 'object_number=45, tag_tid=2, entry_tid=47993' 2026-02-22T19:54:43.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:43.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=44, tag_tid=2, entry_tid=48244' = 'object_number=45, tag_tid=2, entry_tid=47993' 2026-02-22T19:54:43.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=44, tag_tid=2, entry_tid=48244' 2026-02-22T19:54:43.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:43.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:43.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=44, tag_tid=2, entry_tid=48244' 2026-02-22T19:54:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:43.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:43.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:43.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:43.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=44, tag_tid=2, entry_tid=48244' 2026-02-22T19:54:43.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:43.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=48244 2026-02-22T19:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 48244 2026-02-22T19:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:43.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:43.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:43.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:43.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:43.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:43.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:43.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:43.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:43.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:43.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:43.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:43.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:43.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:43.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:43.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:43.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:43.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:43.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:43.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=48490], [object_number=45, tag_tid=2, entry_tid=48489], [object_number=44, tag_tid=2, entry_tid=48488], [object_number=47, tag_tid=2, entry_tid=48487]]], state=connected] 2026-02-22T19:54:43.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:43.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:43.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:43.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:43.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=48490], [object_number=45, tag_tid=2, entry_tid=48489], [object_number=44, tag_tid=2, entry_tid=48488], [object_number=47, tag_tid=2, entry_tid=48487]]], state=connected] 2026-02-22T19:54:43.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=46, tag_tid=2, entry_tid=48490' 2026-02-22T19:54:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=46, tag_tid=2, entry_tid=48490' 2026-02-22T19:54:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=48490' '!=' 'object_number=44, tag_tid=2, entry_tid=48244' 2026-02-22T19:54:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=48490' = 'object_number=44, tag_tid=2, entry_tid=48244' 2026-02-22T19:54:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=46, tag_tid=2, entry_tid=48490' 2026-02-22T19:54:43.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:43.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:43.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:43.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=48490' 2026-02-22T19:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:43.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:43.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:43.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:43.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=48490' 2026-02-22T19:54:43.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:43.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:43.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=48490 2026-02-22T19:54:43.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 48490 2026-02-22T19:54:43.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:43.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:43.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:44.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:44.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:44.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=48725], [object_number=44, tag_tid=2, entry_tid=48724], [object_number=47, tag_tid=2, entry_tid=48723], [object_number=46, tag_tid=2, entry_tid=48722]]], state=connected] 2026-02-22T19:54:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:44.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:44.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:44.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:44.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:44.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:44.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:44.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:44.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=48725], [object_number=44, tag_tid=2, entry_tid=48724], [object_number=47, tag_tid=2, entry_tid=48723], [object_number=46, tag_tid=2, entry_tid=48722]]], state=connected] 2026-02-22T19:54:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=45, tag_tid=2, entry_tid=48725' 2026-02-22T19:54:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=45, tag_tid=2, entry_tid=48725' 2026-02-22T19:54:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=48725' '!=' 'object_number=46, tag_tid=2, entry_tid=48490' 2026-02-22T19:54:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=48725' = 'object_number=46, tag_tid=2, entry_tid=48490' 2026-02-22T19:54:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=45, tag_tid=2, entry_tid=48725' 2026-02-22T19:54:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:44.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:44.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=48725' 2026-02-22T19:54:44.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:44.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:44.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:44.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:44.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=48725' 2026-02-22T19:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=48725 2026-02-22T19:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 48725 2026-02-22T19:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:44.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:44.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:44.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:44.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:44.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:44.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:44.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:44.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=48970], [object_number=45, tag_tid=2, entry_tid=48969], [object_number=44, tag_tid=2, entry_tid=48968], [object_number=47, tag_tid=2, entry_tid=48967]]], state=connected] 2026-02-22T19:54:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:44.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:44.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:44.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=48970], [object_number=45, tag_tid=2, entry_tid=48969], [object_number=44, tag_tid=2, entry_tid=48968], [object_number=47, tag_tid=2, entry_tid=48967]]], state=connected] 2026-02-22T19:54:44.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=46, tag_tid=2, entry_tid=48970' 2026-02-22T19:54:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=46, tag_tid=2, entry_tid=48970' 2026-02-22T19:54:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=48970' '!=' 'object_number=45, tag_tid=2, entry_tid=48725' 2026-02-22T19:54:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=48970' = 'object_number=45, tag_tid=2, entry_tid=48725' 2026-02-22T19:54:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=46, tag_tid=2, entry_tid=48970' 2026-02-22T19:54:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:44.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=48970' 2026-02-22T19:54:44.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:44.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:44.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:44.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:44.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:44.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=48970' 2026-02-22T19:54:44.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:44.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=48970 2026-02-22T19:54:44.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 48970 2026-02-22T19:54:44.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:44.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:44.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:44.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:44.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:44.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:44.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:44.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:44.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:44.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:44.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:44.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:44.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:44.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:44.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:44.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:44.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:44.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:44.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:44.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:44.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:44.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:44.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=49194], [object_number=45, tag_tid=2, entry_tid=49193], [object_number=44, tag_tid=2, entry_tid=49192], [object_number=47, tag_tid=2, entry_tid=49191]]], state=connected] 2026-02-22T19:54:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:44.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:44.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:44.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:44.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:44.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:44.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:44.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:44.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:44.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=49194], [object_number=45, tag_tid=2, entry_tid=49193], [object_number=44, tag_tid=2, entry_tid=49192], [object_number=47, tag_tid=2, entry_tid=49191]]], state=connected] 2026-02-22T19:54:44.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:44.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=46, tag_tid=2, entry_tid=49194' 2026-02-22T19:54:44.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=46, tag_tid=2, entry_tid=49194' 2026-02-22T19:54:44.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=49194' '!=' 'object_number=46, tag_tid=2, entry_tid=48970' 2026-02-22T19:54:44.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=49194' = 'object_number=46, tag_tid=2, entry_tid=48970' 2026-02-22T19:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=46, tag_tid=2, entry_tid=49194' 2026-02-22T19:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:44.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:44.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=49194' 2026-02-22T19:54:44.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:44.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:44.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:44.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:44.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:44.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=49194' 2026-02-22T19:54:44.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:44.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:44.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=49194 2026-02-22T19:54:44.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 49194 2026-02-22T19:54:44.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:44.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:44.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:45.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:45.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:45.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:45.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:45.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:45.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:45.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:45.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:45.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:45.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:45.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:45.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:45.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=49441], [object_number=44, tag_tid=2, entry_tid=49440], [object_number=47, tag_tid=2, entry_tid=49439], [object_number=46, tag_tid=2, entry_tid=49438]]], state=connected] 2026-02-22T19:54:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:45.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:45.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=49441], [object_number=44, tag_tid=2, entry_tid=49440], [object_number=47, tag_tid=2, entry_tid=49439], [object_number=46, tag_tid=2, entry_tid=49438]]], state=connected] 2026-02-22T19:54:45.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=45, tag_tid=2, entry_tid=49441' 2026-02-22T19:54:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=45, tag_tid=2, entry_tid=49441' 2026-02-22T19:54:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=49441' '!=' 'object_number=46, tag_tid=2, entry_tid=49194' 2026-02-22T19:54:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=49441' = 'object_number=46, tag_tid=2, entry_tid=49194' 2026-02-22T19:54:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=45, tag_tid=2, entry_tid=49441' 2026-02-22T19:54:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:45.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=49441' 2026-02-22T19:54:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:45.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=49441' 2026-02-22T19:54:45.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:45.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=49441 2026-02-22T19:54:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 49441 2026-02-22T19:54:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:45.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:45.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:45.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:45.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:45.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:45.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:45.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:45.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:45.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:45.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:45.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:45.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:45.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:45.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:45.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:45.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:45.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=49686], [object_number=45, tag_tid=2, entry_tid=49685], [object_number=44, tag_tid=2, entry_tid=49684], [object_number=47, tag_tid=2, entry_tid=49683]]], state=connected] 2026-02-22T19:54:45.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:45.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:45.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:45.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:45.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=46, tag_tid=2, entry_tid=49686], [object_number=45, tag_tid=2, entry_tid=49685], [object_number=44, tag_tid=2, entry_tid=49684], [object_number=47, tag_tid=2, entry_tid=49683]]], state=connected] 2026-02-22T19:54:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=46, tag_tid=2, entry_tid=49686' 2026-02-22T19:54:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=46, tag_tid=2, entry_tid=49686' 2026-02-22T19:54:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=49686' '!=' 'object_number=45, tag_tid=2, entry_tid=49441' 2026-02-22T19:54:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=46, tag_tid=2, entry_tid=49686' = 'object_number=45, tag_tid=2, entry_tid=49441' 2026-02-22T19:54:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=46, tag_tid=2, entry_tid=49686' 2026-02-22T19:54:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:45.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=49686' 2026-02-22T19:54:45.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:45.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=46, tag_tid=2, entry_tid=49686' 2026-02-22T19:54:45.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:45.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=49686 2026-02-22T19:54:45.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 49686 2026-02-22T19:54:45.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:45.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:45.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:45.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:45.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:45.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:45.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:45.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:45.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:45.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:45.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:45.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:45.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:45.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:45.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:45.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:45.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:45.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=49941], [object_number=44, tag_tid=2, entry_tid=49940], [object_number=47, tag_tid=2, entry_tid=49939], [object_number=46, tag_tid=2, entry_tid=49938]]], state=connected] 2026-02-22T19:54:45.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:45.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:45.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:45.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:45.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:45.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=45, tag_tid=2, entry_tid=49941], [object_number=44, tag_tid=2, entry_tid=49940], [object_number=47, tag_tid=2, entry_tid=49939], [object_number=46, tag_tid=2, entry_tid=49938]]], state=connected] 2026-02-22T19:54:45.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=45, tag_tid=2, entry_tid=49941' 2026-02-22T19:54:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=45, tag_tid=2, entry_tid=49941' 2026-02-22T19:54:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=49941' '!=' 'object_number=46, tag_tid=2, entry_tid=49686' 2026-02-22T19:54:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=45, tag_tid=2, entry_tid=49941' = 'object_number=46, tag_tid=2, entry_tid=49686' 2026-02-22T19:54:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=45, tag_tid=2, entry_tid=49941' 2026-02-22T19:54:45.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:45.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:45.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:45.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=49941' 2026-02-22T19:54:45.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:45.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:45.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:45.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:45.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:45.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=45, tag_tid=2, entry_tid=49941' 2026-02-22T19:54:45.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:45.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:45.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=49941 2026-02-22T19:54:45.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 49941 2026-02-22T19:54:45.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:45.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:45.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:46.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:46.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:46.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:46.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:46.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:46.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:46.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:46.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:46.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:46.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:46.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:46.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:46.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:46.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:46.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:46.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:46.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:46.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:46.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:46.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:46.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:46.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:46.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:46.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:46.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:46.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:46.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:46.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:46.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=44, tag_tid=2, entry_tid=50160], [object_number=47, tag_tid=2, entry_tid=50159], [object_number=46, tag_tid=2, entry_tid=50158], [object_number=45, tag_tid=2, entry_tid=50157]]], state=connected] 2026-02-22T19:54:46.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 11 2026-02-22T19:54:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=44, tag_tid=2, entry_tid=50160], [object_number=47, tag_tid=2, entry_tid=50159], [object_number=46, tag_tid=2, entry_tid=50158], [object_number=45, tag_tid=2, entry_tid=50157]]], state=connected] 2026-02-22T19:54:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=44, tag_tid=2, entry_tid=50160' 2026-02-22T19:54:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=44, tag_tid=2, entry_tid=50160' 2026-02-22T19:54:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=44, tag_tid=2, entry_tid=50160' '!=' 'object_number=45, tag_tid=2, entry_tid=49941' 2026-02-22T19:54:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=44, tag_tid=2, entry_tid=50160' = 'object_number=45, tag_tid=2, entry_tid=49941' 2026-02-22T19:54:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=44, tag_tid=2, entry_tid=50160' 2026-02-22T19:54:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=44, tag_tid=2, entry_tid=50160' 2026-02-22T19:54:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:46.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:46.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:46.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:46.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:46.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=44, tag_tid=2, entry_tid=50160' 2026-02-22T19:54:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=50160 2026-02-22T19:54:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 50160 2026-02-22T19:54:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:46.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:46.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:46.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:46.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:46.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:46.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:46.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:46.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=50313], [object_number=48, tag_tid=2, entry_tid=50312], [object_number=51, tag_tid=2, entry_tid=50311], [object_number=50, tag_tid=2, entry_tid=50310]]], state=connected] 2026-02-22T19:54:46.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:46.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:46.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=50313], [object_number=48, tag_tid=2, entry_tid=50312], [object_number=51, tag_tid=2, entry_tid=50311], [object_number=50, tag_tid=2, entry_tid=50310]]], state=connected] 2026-02-22T19:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=49, tag_tid=2, entry_tid=50313' 2026-02-22T19:54:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=49, tag_tid=2, entry_tid=50313' 2026-02-22T19:54:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=50313' '!=' 'object_number=44, tag_tid=2, entry_tid=50160' 2026-02-22T19:54:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=50313' = 'object_number=44, tag_tid=2, entry_tid=50160' 2026-02-22T19:54:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=49, tag_tid=2, entry_tid=50313' 2026-02-22T19:54:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:46.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:46.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=50313' 2026-02-22T19:54:46.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:46.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:46.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:46.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:46.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:46.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:46.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:46.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=50313' 2026-02-22T19:54:46.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:46.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:46.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=50313 2026-02-22T19:54:46.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 50313 2026-02-22T19:54:46.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:46.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:46.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:46.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:46.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:46.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:46.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:46.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:46.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:46.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:46.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:46.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:47.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=50561], [object_number=48, tag_tid=2, entry_tid=50560], [object_number=51, tag_tid=2, entry_tid=50559], [object_number=50, tag_tid=2, entry_tid=50558]]], state=connected] 2026-02-22T19:54:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:47.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:47.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:47.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:47.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:47.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:47.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:47.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:47.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:47.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:47.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:47.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:47.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=50561], [object_number=48, tag_tid=2, entry_tid=50560], [object_number=51, tag_tid=2, entry_tid=50559], [object_number=50, tag_tid=2, entry_tid=50558]]], state=connected] 2026-02-22T19:54:47.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=49, tag_tid=2, entry_tid=50561' 2026-02-22T19:54:47.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=49, tag_tid=2, entry_tid=50561' 2026-02-22T19:54:47.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=50561' '!=' 'object_number=49, tag_tid=2, entry_tid=50313' 2026-02-22T19:54:47.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:47.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=50561' = 'object_number=49, tag_tid=2, entry_tid=50313' 2026-02-22T19:54:47.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=49, tag_tid=2, entry_tid=50561' 2026-02-22T19:54:47.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:47.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:47.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:47.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=50561' 2026-02-22T19:54:47.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:47.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:47.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:47.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:47.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:47.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=50561' 2026-02-22T19:54:47.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:47.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=50561 2026-02-22T19:54:47.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 50561 2026-02-22T19:54:47.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:47.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:47.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:47.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:47.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:47.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:47.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:47.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:47.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:47.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:47.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:47.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:47.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:47.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:47.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:47.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:47.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:47.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:47.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:47.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:47.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:47.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:47.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:47.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:47.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:47.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:47.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=50832], [object_number=51, tag_tid=2, entry_tid=50831], [object_number=50, tag_tid=2, entry_tid=50830], [object_number=49, tag_tid=2, entry_tid=50829]]], state=connected] 2026-02-22T19:54:47.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:47.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:47.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:47.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:47.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:47.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:47.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:47.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:47.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:47.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:47.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:47.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:47.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=50832], [object_number=51, tag_tid=2, entry_tid=50831], [object_number=50, tag_tid=2, entry_tid=50830], [object_number=49, tag_tid=2, entry_tid=50829]]], state=connected] 2026-02-22T19:54:47.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=48, tag_tid=2, entry_tid=50832' 2026-02-22T19:54:47.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=48, tag_tid=2, entry_tid=50832' 2026-02-22T19:54:47.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=50832' '!=' 'object_number=49, tag_tid=2, entry_tid=50561' 2026-02-22T19:54:47.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:47.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=50832' = 'object_number=49, tag_tid=2, entry_tid=50561' 2026-02-22T19:54:47.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=48, tag_tid=2, entry_tid=50832' 2026-02-22T19:54:47.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:47.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:47.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=50832' 2026-02-22T19:54:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:47.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:47.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:47.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=50832' 2026-02-22T19:54:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:47.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=50832 2026-02-22T19:54:47.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 50832 2026-02-22T19:54:47.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:47.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:47.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:47.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:47.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:47.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:47.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:47.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:47.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:47.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:47.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:47.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:47.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:47.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=51109], [object_number=48, tag_tid=2, entry_tid=51108], [object_number=51, tag_tid=2, entry_tid=51107], [object_number=50, tag_tid=2, entry_tid=51106]]], state=connected] 2026-02-22T19:54:47.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:47.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:47.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:47.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:47.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:47.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:47.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:47.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:47.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=51109], [object_number=48, tag_tid=2, entry_tid=51108], [object_number=51, tag_tid=2, entry_tid=51107], [object_number=50, tag_tid=2, entry_tid=51106]]], state=connected] 2026-02-22T19:54:47.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:47.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=49, tag_tid=2, entry_tid=51109' 2026-02-22T19:54:47.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=49, tag_tid=2, entry_tid=51109' 2026-02-22T19:54:47.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=51109' '!=' 'object_number=48, tag_tid=2, entry_tid=50832' 2026-02-22T19:54:47.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:47.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=51109' = 'object_number=48, tag_tid=2, entry_tid=50832' 2026-02-22T19:54:47.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=49, tag_tid=2, entry_tid=51109' 2026-02-22T19:54:47.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:47.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:47.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:47.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=51109' 2026-02-22T19:54:47.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:47.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:47.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:47.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:47.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=51109' 2026-02-22T19:54:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:47.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=51109 2026-02-22T19:54:47.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 51109 2026-02-22T19:54:47.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:47.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:47.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:47.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:47.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:47.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:47.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:47.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:47.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:47.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:47.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:48.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:48.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:48.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:48.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:48.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:48.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:48.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:48.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:48.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:48.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:48.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:48.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:48.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:48.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:48.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:48.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:48.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=51, tag_tid=2, entry_tid=51399], [object_number=50, tag_tid=2, entry_tid=51398], [object_number=49, tag_tid=2, entry_tid=51397], [object_number=48, tag_tid=2, entry_tid=51396]]], state=connected] 2026-02-22T19:54:48.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:48.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:48.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:48.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:48.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:48.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=51, tag_tid=2, entry_tid=51399], [object_number=50, tag_tid=2, entry_tid=51398], [object_number=49, tag_tid=2, entry_tid=51397], [object_number=48, tag_tid=2, entry_tid=51396]]], state=connected] 2026-02-22T19:54:48.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=51, tag_tid=2, entry_tid=51399' 2026-02-22T19:54:48.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=51, tag_tid=2, entry_tid=51399' 2026-02-22T19:54:48.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=51, tag_tid=2, entry_tid=51399' '!=' 'object_number=49, tag_tid=2, entry_tid=51109' 2026-02-22T19:54:48.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:48.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=51, tag_tid=2, entry_tid=51399' = 'object_number=49, tag_tid=2, entry_tid=51109' 2026-02-22T19:54:48.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=51, tag_tid=2, entry_tid=51399' 2026-02-22T19:54:48.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:48.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:48.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:48.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=51, tag_tid=2, entry_tid=51399' 2026-02-22T19:54:48.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:48.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=51, tag_tid=2, entry_tid=51399' 2026-02-22T19:54:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=51399 2026-02-22T19:54:48.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 51399 2026-02-22T19:54:48.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:48.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:48.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:48.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:48.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:48.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:48.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:48.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:48.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:48.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:48.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:48.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:48.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:48.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:48.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:48.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:48.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:48.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:48.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:48.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:48.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:48.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=51648], [object_number=51, tag_tid=2, entry_tid=51647], [object_number=50, tag_tid=2, entry_tid=51646], [object_number=49, tag_tid=2, entry_tid=51645]]], state=connected] 2026-02-22T19:54:48.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:48.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:48.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:48.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:48.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:48.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=51648], [object_number=51, tag_tid=2, entry_tid=51647], [object_number=50, tag_tid=2, entry_tid=51646], [object_number=49, tag_tid=2, entry_tid=51645]]], state=connected] 2026-02-22T19:54:48.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=48, tag_tid=2, entry_tid=51648' 2026-02-22T19:54:48.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=48, tag_tid=2, entry_tid=51648' 2026-02-22T19:54:48.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=51648' '!=' 'object_number=51, tag_tid=2, entry_tid=51399' 2026-02-22T19:54:48.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:48.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=51648' = 'object_number=51, tag_tid=2, entry_tid=51399' 2026-02-22T19:54:48.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=48, tag_tid=2, entry_tid=51648' 2026-02-22T19:54:48.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:48.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:48.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:48.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=51648' 2026-02-22T19:54:48.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:48.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:48.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:48.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:48.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:48.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=51648' 2026-02-22T19:54:48.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:48.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=51648 2026-02-22T19:54:48.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 51648 2026-02-22T19:54:48.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:48.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:48.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:48.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:48.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:48.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:48.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:48.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:48.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:48.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:48.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:48.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:48.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:48.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:48.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:48.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:48.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:48.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:48.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:48.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:48.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:48.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:48.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:48.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:48.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:48.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:48.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:48.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:48.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=51980], [object_number=51, tag_tid=2, entry_tid=51979], [object_number=50, tag_tid=2, entry_tid=51978], [object_number=49, tag_tid=2, entry_tid=51977]]], state=connected] 2026-02-22T19:54:48.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=51980], [object_number=51, tag_tid=2, entry_tid=51979], [object_number=50, tag_tid=2, entry_tid=51978], [object_number=49, tag_tid=2, entry_tid=51977]]], state=connected] 2026-02-22T19:54:48.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=48, tag_tid=2, entry_tid=51980' 2026-02-22T19:54:48.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=48, tag_tid=2, entry_tid=51980' 2026-02-22T19:54:48.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=51980' '!=' 'object_number=48, tag_tid=2, entry_tid=51648' 2026-02-22T19:54:48.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:48.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=51980' = 'object_number=48, tag_tid=2, entry_tid=51648' 2026-02-22T19:54:48.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=48, tag_tid=2, entry_tid=51980' 2026-02-22T19:54:48.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:48.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:48.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=51980' 2026-02-22T19:54:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:48.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:48.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:48.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=51980' 2026-02-22T19:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=51980 2026-02-22T19:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 51980 2026-02-22T19:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:49.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:49.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:49.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:49.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:49.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:49.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:49.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:49.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:49.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:49.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:49.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:49.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=2, entry_tid=52258], [object_number=49, tag_tid=2, entry_tid=52257], [object_number=48, tag_tid=2, entry_tid=52256], [object_number=51, tag_tid=2, entry_tid=52255]]], state=connected] 2026-02-22T19:54:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:49.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:49.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:49.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:49.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=2, entry_tid=52258], [object_number=49, tag_tid=2, entry_tid=52257], [object_number=48, tag_tid=2, entry_tid=52256], [object_number=51, tag_tid=2, entry_tid=52255]]], state=connected] 2026-02-22T19:54:49.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=50, tag_tid=2, entry_tid=52258' 2026-02-22T19:54:49.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=50, tag_tid=2, entry_tid=52258' 2026-02-22T19:54:49.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=2, entry_tid=52258' '!=' 'object_number=48, tag_tid=2, entry_tid=51980' 2026-02-22T19:54:49.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:49.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=2, entry_tid=52258' = 'object_number=48, tag_tid=2, entry_tid=51980' 2026-02-22T19:54:49.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=50, tag_tid=2, entry_tid=52258' 2026-02-22T19:54:49.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:49.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:49.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:49.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=2, entry_tid=52258' 2026-02-22T19:54:49.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:49.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:49.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:49.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:49.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:49.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=2, entry_tid=52258' 2026-02-22T19:54:49.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:49.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=52258 2026-02-22T19:54:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 52258 2026-02-22T19:54:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:49.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:49.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:49.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:49.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:49.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:49.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:49.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:49.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:49.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:49.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:49.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:49.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:49.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:49.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:49.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:49.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:49.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:49.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:49.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:49.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:49.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:49.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:49.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:49.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:49.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:49.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:49.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:49.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:49.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:49.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=2, entry_tid=52514], [object_number=49, tag_tid=2, entry_tid=52513], [object_number=48, tag_tid=2, entry_tid=52512], [object_number=51, tag_tid=2, entry_tid=52511]]], state=connected] 2026-02-22T19:54:49.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:49.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:49.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:49.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:49.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:49.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=2, entry_tid=52514], [object_number=49, tag_tid=2, entry_tid=52513], [object_number=48, tag_tid=2, entry_tid=52512], [object_number=51, tag_tid=2, entry_tid=52511]]], state=connected] 2026-02-22T19:54:49.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=50, tag_tid=2, entry_tid=52514' 2026-02-22T19:54:49.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=50, tag_tid=2, entry_tid=52514' 2026-02-22T19:54:49.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=2, entry_tid=52514' '!=' 'object_number=50, tag_tid=2, entry_tid=52258' 2026-02-22T19:54:49.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:49.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=2, entry_tid=52514' = 'object_number=50, tag_tid=2, entry_tid=52258' 2026-02-22T19:54:49.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=50, tag_tid=2, entry_tid=52514' 2026-02-22T19:54:49.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:49.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:49.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:49.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=2, entry_tid=52514' 2026-02-22T19:54:49.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:49.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:49.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=2, entry_tid=52514' 2026-02-22T19:54:49.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:49.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=52514 2026-02-22T19:54:49.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 52514 2026-02-22T19:54:49.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:49.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:49.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:49.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:49.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:49.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:49.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:49.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:49.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:49.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:49.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:49.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:49.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:49.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:49.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:49.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:49.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:49.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:49.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=52773], [object_number=48, tag_tid=2, entry_tid=52772], [object_number=51, tag_tid=2, entry_tid=52771], [object_number=50, tag_tid=2, entry_tid=52770]]], state=connected] 2026-02-22T19:54:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=52773], [object_number=48, tag_tid=2, entry_tid=52772], [object_number=51, tag_tid=2, entry_tid=52771], [object_number=50, tag_tid=2, entry_tid=52770]]], state=connected] 2026-02-22T19:54:49.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:49.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=49, tag_tid=2, entry_tid=52773' 2026-02-22T19:54:49.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=49, tag_tid=2, entry_tid=52773' 2026-02-22T19:54:49.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=52773' '!=' 'object_number=50, tag_tid=2, entry_tid=52514' 2026-02-22T19:54:49.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:49.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=52773' = 'object_number=50, tag_tid=2, entry_tid=52514' 2026-02-22T19:54:49.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=49, tag_tid=2, entry_tid=52773' 2026-02-22T19:54:49.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:49.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:49.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:49.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=52773' 2026-02-22T19:54:49.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:49.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:49.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:49.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:49.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=52773' 2026-02-22T19:54:49.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:49.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=52773 2026-02-22T19:54:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 52773 2026-02-22T19:54:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:50.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:50.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:50.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:50.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:50.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:50.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:50.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:50.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=53037], [object_number=48, tag_tid=2, entry_tid=53036], [object_number=51, tag_tid=2, entry_tid=53035], [object_number=50, tag_tid=2, entry_tid=53034]]], state=connected] 2026-02-22T19:54:50.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:50.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:50.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:50.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:50.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:50.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:50.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:50.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:50.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:50.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=53037], [object_number=48, tag_tid=2, entry_tid=53036], [object_number=51, tag_tid=2, entry_tid=53035], [object_number=50, tag_tid=2, entry_tid=53034]]], state=connected] 2026-02-22T19:54:50.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=49, tag_tid=2, entry_tid=53037' 2026-02-22T19:54:50.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=49, tag_tid=2, entry_tid=53037' 2026-02-22T19:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=53037' '!=' 'object_number=49, tag_tid=2, entry_tid=52773' 2026-02-22T19:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=49, tag_tid=2, entry_tid=53037' = 'object_number=49, tag_tid=2, entry_tid=52773' 2026-02-22T19:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=49, tag_tid=2, entry_tid=53037' 2026-02-22T19:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:50.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:50.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=53037' 2026-02-22T19:54:50.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:50.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:50.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:50.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:50.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:50.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:50.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=53037' 2026-02-22T19:54:50.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:50.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:50.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=53037 2026-02-22T19:54:50.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 53037 2026-02-22T19:54:50.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:50.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:50.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:50.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:50.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:50.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:50.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:50.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:50.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:50.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:50.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:50.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:50.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:50.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:50.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:50.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:50.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:50.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:50.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:50.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:50.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:50.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=2, entry_tid=53286], [object_number=49, tag_tid=2, entry_tid=53285], [object_number=48, tag_tid=2, entry_tid=53284], [object_number=51, tag_tid=2, entry_tid=53283]]], state=connected] 2026-02-22T19:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:50.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:50.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=2, entry_tid=53286], [object_number=49, tag_tid=2, entry_tid=53285], [object_number=48, tag_tid=2, entry_tid=53284], [object_number=51, tag_tid=2, entry_tid=53283]]], state=connected] 2026-02-22T19:54:50.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=50, tag_tid=2, entry_tid=53286' 2026-02-22T19:54:50.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=50, tag_tid=2, entry_tid=53286' 2026-02-22T19:54:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=2, entry_tid=53286' '!=' 'object_number=49, tag_tid=2, entry_tid=53037' 2026-02-22T19:54:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=2, entry_tid=53286' = 'object_number=49, tag_tid=2, entry_tid=53037' 2026-02-22T19:54:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=50, tag_tid=2, entry_tid=53286' 2026-02-22T19:54:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:50.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:50.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:50.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:50.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=2, entry_tid=53286' 2026-02-22T19:54:50.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:50.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:50.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:50.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:50.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:50.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:50.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=2, entry_tid=53286' 2026-02-22T19:54:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=53286 2026-02-22T19:54:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 53286 2026-02-22T19:54:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:50.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:50.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:50.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:50.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:50.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:50.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:50.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:50.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:50.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:50.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:50.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:50.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:50.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:50.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:50.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:50.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:50.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=2, entry_tid=53566], [object_number=49, tag_tid=2, entry_tid=53565], [object_number=48, tag_tid=2, entry_tid=53564], [object_number=51, tag_tid=2, entry_tid=53563]]], state=connected] 2026-02-22T19:54:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:51.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:51.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:51.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:51.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:51.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:51.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:51.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:51.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=2, entry_tid=53566], [object_number=49, tag_tid=2, entry_tid=53565], [object_number=48, tag_tid=2, entry_tid=53564], [object_number=51, tag_tid=2, entry_tid=53563]]], state=connected] 2026-02-22T19:54:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=50, tag_tid=2, entry_tid=53566' 2026-02-22T19:54:51.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=50, tag_tid=2, entry_tid=53566' 2026-02-22T19:54:51.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=2, entry_tid=53566' '!=' 'object_number=50, tag_tid=2, entry_tid=53286' 2026-02-22T19:54:51.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:51.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=2, entry_tid=53566' = 'object_number=50, tag_tid=2, entry_tid=53286' 2026-02-22T19:54:51.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=50, tag_tid=2, entry_tid=53566' 2026-02-22T19:54:51.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:51.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:51.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:51.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=2, entry_tid=53566' 2026-02-22T19:54:51.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:51.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:51.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:51.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:51.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=2, entry_tid=53566' 2026-02-22T19:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:51.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=53566 2026-02-22T19:54:51.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 53566 2026-02-22T19:54:51.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:51.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:51.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:51.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:51.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:51.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:51.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:51.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:51.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:51.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:51.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:51.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:51.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:51.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:51.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:51.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:51.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:51.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:51.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=53844], [object_number=51, tag_tid=2, entry_tid=53843], [object_number=50, tag_tid=2, entry_tid=53842], [object_number=49, tag_tid=2, entry_tid=53841]]], state=connected] 2026-02-22T19:54:51.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:51.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:51.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:51.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:51.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:51.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=53844], [object_number=51, tag_tid=2, entry_tid=53843], [object_number=50, tag_tid=2, entry_tid=53842], [object_number=49, tag_tid=2, entry_tid=53841]]], state=connected] 2026-02-22T19:54:51.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=48, tag_tid=2, entry_tid=53844' 2026-02-22T19:54:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=48, tag_tid=2, entry_tid=53844' 2026-02-22T19:54:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=53844' '!=' 'object_number=50, tag_tid=2, entry_tid=53566' 2026-02-22T19:54:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=53844' = 'object_number=50, tag_tid=2, entry_tid=53566' 2026-02-22T19:54:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=48, tag_tid=2, entry_tid=53844' 2026-02-22T19:54:51.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:51.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:51.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:51.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=53844' 2026-02-22T19:54:51.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:51.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:51.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:51.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:51.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:51.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=53844' 2026-02-22T19:54:51.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:51.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=53844 2026-02-22T19:54:51.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 53844 2026-02-22T19:54:51.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:51.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:51.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:51.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:51.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:51.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:51.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:51.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:51.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:51.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:51.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:51.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:51.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:51.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:51.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:51.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:51.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:51.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:51.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:51.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:51.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:51.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:51.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:51.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:51.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:51.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:51.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:51.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:51.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:51.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:51.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:51.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=54128], [object_number=51, tag_tid=2, entry_tid=54127], [object_number=50, tag_tid=2, entry_tid=54126], [object_number=49, tag_tid=2, entry_tid=54125]]], state=connected] 2026-02-22T19:54:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:51.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:51.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:51.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:51.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=48, tag_tid=2, entry_tid=54128], [object_number=51, tag_tid=2, entry_tid=54127], [object_number=50, tag_tid=2, entry_tid=54126], [object_number=49, tag_tid=2, entry_tid=54125]]], state=connected] 2026-02-22T19:54:51.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=48, tag_tid=2, entry_tid=54128' 2026-02-22T19:54:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=48, tag_tid=2, entry_tid=54128' 2026-02-22T19:54:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=54128' '!=' 'object_number=48, tag_tid=2, entry_tid=53844' 2026-02-22T19:54:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=48, tag_tid=2, entry_tid=54128' = 'object_number=48, tag_tid=2, entry_tid=53844' 2026-02-22T19:54:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=48, tag_tid=2, entry_tid=54128' 2026-02-22T19:54:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:51.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=54128' 2026-02-22T19:54:51.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:51.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:51.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:51.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:51.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=48, tag_tid=2, entry_tid=54128' 2026-02-22T19:54:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:51.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=54128 2026-02-22T19:54:51.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 54128 2026-02-22T19:54:51.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:51.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:51.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:52.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:52.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:52.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:52.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:52.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:52.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:52.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:52.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:52.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:52.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:52.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:52.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:52.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=54338], [object_number=53, tag_tid=2, entry_tid=54337], [object_number=52, tag_tid=2, entry_tid=54336], [object_number=51, tag_tid=2, entry_tid=54335]]], state=connected] 2026-02-22T19:54:52.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:52.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T19:54:52.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:52.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:52.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:52.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=54338], [object_number=53, tag_tid=2, entry_tid=54337], [object_number=52, tag_tid=2, entry_tid=54336], [object_number=51, tag_tid=2, entry_tid=54335]]], state=connected] 2026-02-22T19:54:52.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=54, tag_tid=2, entry_tid=54338' 2026-02-22T19:54:52.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=54, tag_tid=2, entry_tid=54338' 2026-02-22T19:54:52.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=54338' '!=' 'object_number=48, tag_tid=2, entry_tid=54128' 2026-02-22T19:54:52.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:52.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=54338' = 'object_number=48, tag_tid=2, entry_tid=54128' 2026-02-22T19:54:52.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=54, tag_tid=2, entry_tid=54338' 2026-02-22T19:54:52.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:52.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:52.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:52.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=54338' 2026-02-22T19:54:52.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:52.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:52.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:52.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:52.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:52.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=54338' 2026-02-22T19:54:52.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:52.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=54338 2026-02-22T19:54:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 54338 2026-02-22T19:54:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:52.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:52.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:52.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:52.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:52.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:52.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:52.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:52.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:52.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:52.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:52.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:52.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:52.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:52.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:52.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:52.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:52.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:52.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:52.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=54574], [object_number=53, tag_tid=2, entry_tid=54573], [object_number=52, tag_tid=2, entry_tid=54572], [object_number=55, tag_tid=2, entry_tid=54571]]], state=connected] 2026-02-22T19:54:52.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:52.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:52.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:52.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:52.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:52.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:52.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=54574], [object_number=53, tag_tid=2, entry_tid=54573], [object_number=52, tag_tid=2, entry_tid=54572], [object_number=55, tag_tid=2, entry_tid=54571]]], state=connected] 2026-02-22T19:54:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=54, tag_tid=2, entry_tid=54574' 2026-02-22T19:54:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=54, tag_tid=2, entry_tid=54574' 2026-02-22T19:54:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=54574' '!=' 'object_number=54, tag_tid=2, entry_tid=54338' 2026-02-22T19:54:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=54574' = 'object_number=54, tag_tid=2, entry_tid=54338' 2026-02-22T19:54:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=54, tag_tid=2, entry_tid=54574' 2026-02-22T19:54:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:52.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=54574' 2026-02-22T19:54:52.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:52.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:52.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:52.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:52.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:52.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=54574' 2026-02-22T19:54:52.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:52.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=54574 2026-02-22T19:54:52.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 54574 2026-02-22T19:54:52.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:52.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:52.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:52.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:52.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:52.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:52.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:52.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:52.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:52.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:52.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:52.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:52.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:52.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:52.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=54810], [object_number=53, tag_tid=2, entry_tid=54809], [object_number=52, tag_tid=2, entry_tid=54808], [object_number=55, tag_tid=2, entry_tid=54807]]], state=connected] 2026-02-22T19:54:52.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:52.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:52.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:52.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=54810], [object_number=53, tag_tid=2, entry_tid=54809], [object_number=52, tag_tid=2, entry_tid=54808], [object_number=55, tag_tid=2, entry_tid=54807]]], state=connected] 2026-02-22T19:54:52.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=54, tag_tid=2, entry_tid=54810' 2026-02-22T19:54:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=54, tag_tid=2, entry_tid=54810' 2026-02-22T19:54:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=54810' '!=' 'object_number=54, tag_tid=2, entry_tid=54574' 2026-02-22T19:54:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=54810' = 'object_number=54, tag_tid=2, entry_tid=54574' 2026-02-22T19:54:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=54, tag_tid=2, entry_tid=54810' 2026-02-22T19:54:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:52.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=54810' 2026-02-22T19:54:52.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:52.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:52.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:52.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:52.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:52.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=54810' 2026-02-22T19:54:52.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:52.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:52.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=54810 2026-02-22T19:54:52.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 54810 2026-02-22T19:54:52.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:52.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:52.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:53.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:53.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:53.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:53.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:53.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:53.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:53.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:53.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:53.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:53.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:53.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:53.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:53.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:53.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:53.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:53.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:53.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:53.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=53, tag_tid=2, entry_tid=55029], [object_number=52, tag_tid=2, entry_tid=55028], [object_number=55, tag_tid=2, entry_tid=55027], [object_number=54, tag_tid=2, entry_tid=55026]]], state=connected] 2026-02-22T19:54:53.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:53.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:53.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:53.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:53.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:53.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:53.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:53.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:53.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:53.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:53.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:53.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:53.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=53, tag_tid=2, entry_tid=55029], [object_number=52, tag_tid=2, entry_tid=55028], [object_number=55, tag_tid=2, entry_tid=55027], [object_number=54, tag_tid=2, entry_tid=55026]]], state=connected] 2026-02-22T19:54:53.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=53, tag_tid=2, entry_tid=55029' 2026-02-22T19:54:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=53, tag_tid=2, entry_tid=55029' 2026-02-22T19:54:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=53, tag_tid=2, entry_tid=55029' '!=' 'object_number=54, tag_tid=2, entry_tid=54810' 2026-02-22T19:54:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=53, tag_tid=2, entry_tid=55029' = 'object_number=54, tag_tid=2, entry_tid=54810' 2026-02-22T19:54:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=53, tag_tid=2, entry_tid=55029' 2026-02-22T19:54:53.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:53.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:53.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:53.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=53, tag_tid=2, entry_tid=55029' 2026-02-22T19:54:53.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:53.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:53.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=53, tag_tid=2, entry_tid=55029' 2026-02-22T19:54:53.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:53.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=55029 2026-02-22T19:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 55029 2026-02-22T19:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:53.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:53.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:53.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:53.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:53.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:53.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=55248], [object_number=55, tag_tid=2, entry_tid=55247], [object_number=54, tag_tid=2, entry_tid=55246], [object_number=53, tag_tid=2, entry_tid=55245]]], state=connected] 2026-02-22T19:54:53.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:53.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:53.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:53.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:53.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:53.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:53.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:53.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:53.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=55248], [object_number=55, tag_tid=2, entry_tid=55247], [object_number=54, tag_tid=2, entry_tid=55246], [object_number=53, tag_tid=2, entry_tid=55245]]], state=connected] 2026-02-22T19:54:53.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=52, tag_tid=2, entry_tid=55248' 2026-02-22T19:54:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=52, tag_tid=2, entry_tid=55248' 2026-02-22T19:54:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=55248' '!=' 'object_number=53, tag_tid=2, entry_tid=55029' 2026-02-22T19:54:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=55248' = 'object_number=53, tag_tid=2, entry_tid=55029' 2026-02-22T19:54:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=52, tag_tid=2, entry_tid=55248' 2026-02-22T19:54:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:53.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=55248' 2026-02-22T19:54:53.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:53.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=55248' 2026-02-22T19:54:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=55248 2026-02-22T19:54:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 55248 2026-02-22T19:54:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:53.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:53.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:53.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:53.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:53.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:53.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:53.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:53.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:53.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:53.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:53.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:53.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:53.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:53.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=55480], [object_number=55, tag_tid=2, entry_tid=55479], [object_number=54, tag_tid=2, entry_tid=55478], [object_number=53, tag_tid=2, entry_tid=55477]]], state=connected] 2026-02-22T19:54:53.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:53.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=55480], [object_number=55, tag_tid=2, entry_tid=55479], [object_number=54, tag_tid=2, entry_tid=55478], [object_number=53, tag_tid=2, entry_tid=55477]]], state=connected] 2026-02-22T19:54:53.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:53.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=52, tag_tid=2, entry_tid=55480' 2026-02-22T19:54:53.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=52, tag_tid=2, entry_tid=55480' 2026-02-22T19:54:53.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=55480' '!=' 'object_number=52, tag_tid=2, entry_tid=55248' 2026-02-22T19:54:53.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:53.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=55480' = 'object_number=52, tag_tid=2, entry_tid=55248' 2026-02-22T19:54:53.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=52, tag_tid=2, entry_tid=55480' 2026-02-22T19:54:53.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:53.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:53.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:53.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=55480' 2026-02-22T19:54:53.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:53.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:53.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:53.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:53.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:53.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=55480' 2026-02-22T19:54:53.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:53.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=55480 2026-02-22T19:54:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 55480 2026-02-22T19:54:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:53.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:53.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:54.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:54.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:54.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:54.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:54.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:54.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:54.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:54.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:54.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:54.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:54.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:54.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:54.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:54.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:54.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:54.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:54.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:54.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:54.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:54.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:54.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:54.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:54.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:54.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:54.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:54.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:54.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:54.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:54.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:54.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:54.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:54.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:54.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:54.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:54.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:54.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=55770], [object_number=53, tag_tid=2, entry_tid=55769], [object_number=52, tag_tid=2, entry_tid=55768], [object_number=55, tag_tid=2, entry_tid=55767]]], state=connected] 2026-02-22T19:54:54.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:54.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:54.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:54.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:54.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:54.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:54.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:54.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=55770], [object_number=53, tag_tid=2, entry_tid=55769], [object_number=52, tag_tid=2, entry_tid=55768], [object_number=55, tag_tid=2, entry_tid=55767]]], state=connected] 2026-02-22T19:54:54.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=54, tag_tid=2, entry_tid=55770' 2026-02-22T19:54:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=54, tag_tid=2, entry_tid=55770' 2026-02-22T19:54:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=55770' '!=' 'object_number=52, tag_tid=2, entry_tid=55480' 2026-02-22T19:54:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=55770' = 'object_number=52, tag_tid=2, entry_tid=55480' 2026-02-22T19:54:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=54, tag_tid=2, entry_tid=55770' 2026-02-22T19:54:54.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:54.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:54.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:54.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:54.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=55770' 2026-02-22T19:54:54.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:54.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:54.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:54.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=55770' 2026-02-22T19:54:54.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:54.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:54.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=55770 2026-02-22T19:54:54.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 55770 2026-02-22T19:54:54.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:54.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:54.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:54.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:54.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:54.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:54.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:54.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:54.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:54.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:54.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:54.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:54.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:54.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:54.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:54.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:54.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:54.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:54.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:54.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:54.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:54.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:54.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:54.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=56020], [object_number=55, tag_tid=2, entry_tid=56019], [object_number=54, tag_tid=2, entry_tid=56018], [object_number=53, tag_tid=2, entry_tid=56017]]], state=connected] 2026-02-22T19:54:54.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:54.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:54.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:54.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:54.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:54.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:54.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:54.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:54.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:54.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:54.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:54.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:54.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:54.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:54.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:54.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:54.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:54.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:54.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=56020], [object_number=55, tag_tid=2, entry_tid=56019], [object_number=54, tag_tid=2, entry_tid=56018], [object_number=53, tag_tid=2, entry_tid=56017]]], state=connected] 2026-02-22T19:54:54.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=52, tag_tid=2, entry_tid=56020' 2026-02-22T19:54:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=52, tag_tid=2, entry_tid=56020' 2026-02-22T19:54:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=56020' '!=' 'object_number=54, tag_tid=2, entry_tid=55770' 2026-02-22T19:54:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=56020' = 'object_number=54, tag_tid=2, entry_tid=55770' 2026-02-22T19:54:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=52, tag_tid=2, entry_tid=56020' 2026-02-22T19:54:54.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:54.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:54.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:54.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=56020' 2026-02-22T19:54:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:54.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:54.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:54.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:54.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=56020' 2026-02-22T19:54:54.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:54.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:54.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=56020 2026-02-22T19:54:54.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 56020 2026-02-22T19:54:54.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:54.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:54.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:54.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:54.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:54.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:54.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:55.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:55.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:55.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:55.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:55.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:55.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:55.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=56270], [object_number=53, tag_tid=2, entry_tid=56269], [object_number=52, tag_tid=2, entry_tid=56268], [object_number=55, tag_tid=2, entry_tid=56267]]], state=connected] 2026-02-22T19:54:55.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:55.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:55.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:55.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:55.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:55.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:55.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=56270], [object_number=53, tag_tid=2, entry_tid=56269], [object_number=52, tag_tid=2, entry_tid=56268], [object_number=55, tag_tid=2, entry_tid=56267]]], state=connected] 2026-02-22T19:54:55.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=54, tag_tid=2, entry_tid=56270' 2026-02-22T19:54:55.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=54, tag_tid=2, entry_tid=56270' 2026-02-22T19:54:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=56270' '!=' 'object_number=52, tag_tid=2, entry_tid=56020' 2026-02-22T19:54:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=56270' = 'object_number=52, tag_tid=2, entry_tid=56020' 2026-02-22T19:54:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=54, tag_tid=2, entry_tid=56270' 2026-02-22T19:54:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:55.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:55.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=56270' 2026-02-22T19:54:55.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=56270' 2026-02-22T19:54:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=56270 2026-02-22T19:54:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 56270 2026-02-22T19:54:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:55.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:55.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:55.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:55.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:55.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:55.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:55.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:55.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:55.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:55.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:55.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:55.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:55.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:55.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:55.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:55.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:55.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:55.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:55.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:55.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:55.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:55.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:55.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:55.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:55.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=53, tag_tid=2, entry_tid=56509], [object_number=52, tag_tid=2, entry_tid=56508], [object_number=55, tag_tid=2, entry_tid=56507], [object_number=54, tag_tid=2, entry_tid=56506]]], state=connected] 2026-02-22T19:54:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:55.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:55.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:55.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:55.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:55.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:55.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:55.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:55.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=53, tag_tid=2, entry_tid=56509], [object_number=52, tag_tid=2, entry_tid=56508], [object_number=55, tag_tid=2, entry_tid=56507], [object_number=54, tag_tid=2, entry_tid=56506]]], state=connected] 2026-02-22T19:54:55.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=53, tag_tid=2, entry_tid=56509' 2026-02-22T19:54:55.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=53, tag_tid=2, entry_tid=56509' 2026-02-22T19:54:55.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=53, tag_tid=2, entry_tid=56509' '!=' 'object_number=54, tag_tid=2, entry_tid=56270' 2026-02-22T19:54:55.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:55.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=53, tag_tid=2, entry_tid=56509' = 'object_number=54, tag_tid=2, entry_tid=56270' 2026-02-22T19:54:55.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=53, tag_tid=2, entry_tid=56509' 2026-02-22T19:54:55.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:55.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:55.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=53, tag_tid=2, entry_tid=56509' 2026-02-22T19:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:55.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:55.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:55.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:55.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=53, tag_tid=2, entry_tid=56509' 2026-02-22T19:54:55.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:55.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=56509 2026-02-22T19:54:55.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 56509 2026-02-22T19:54:55.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:55.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:55.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:55.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:55.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:55.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:55.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:55.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:55.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:55.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:55.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:55.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:55.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:55.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:55.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:55.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:55.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:55.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:55.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=56758], [object_number=53, tag_tid=2, entry_tid=56757], [object_number=52, tag_tid=2, entry_tid=56756], [object_number=55, tag_tid=2, entry_tid=56755]]], state=connected] 2026-02-22T19:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:55.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:55.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:55.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:55.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:55.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:55.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:55.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:55.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:55.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:55.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:55.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=54, tag_tid=2, entry_tid=56758], [object_number=53, tag_tid=2, entry_tid=56757], [object_number=52, tag_tid=2, entry_tid=56756], [object_number=55, tag_tid=2, entry_tid=56755]]], state=connected] 2026-02-22T19:54:55.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=54, tag_tid=2, entry_tid=56758' 2026-02-22T19:54:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=54, tag_tid=2, entry_tid=56758' 2026-02-22T19:54:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=56758' '!=' 'object_number=53, tag_tid=2, entry_tid=56509' 2026-02-22T19:54:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=54, tag_tid=2, entry_tid=56758' = 'object_number=53, tag_tid=2, entry_tid=56509' 2026-02-22T19:54:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=54, tag_tid=2, entry_tid=56758' 2026-02-22T19:54:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:55.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=56758' 2026-02-22T19:54:55.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:55.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:55.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:55.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=54, tag_tid=2, entry_tid=56758' 2026-02-22T19:54:55.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:55.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=56758 2026-02-22T19:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 56758 2026-02-22T19:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:55.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:55.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:56.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:56.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:56.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:56.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:56.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:56.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:56.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:56.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:56.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:56.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:56.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:56.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:56.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:56.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:56.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:56.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:56.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:56.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:56.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:56.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:56.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:56.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:56.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:56.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:56.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:56.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:56.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:56.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:56.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:56.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:56.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:56.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:56.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:56.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:56.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:56.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:56.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:56.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:56.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:56.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:56.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:56.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:56.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:56.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:56.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:56.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=55, tag_tid=2, entry_tid=57031], [object_number=54, tag_tid=2, entry_tid=57030], [object_number=53, tag_tid=2, entry_tid=57029], [object_number=52, tag_tid=2, entry_tid=57028]]], state=connected] 2026-02-22T19:54:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:56.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:56.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:56.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:56.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:56.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=55, tag_tid=2, entry_tid=57031], [object_number=54, tag_tid=2, entry_tid=57030], [object_number=53, tag_tid=2, entry_tid=57029], [object_number=52, tag_tid=2, entry_tid=57028]]], state=connected] 2026-02-22T19:54:56.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=55, tag_tid=2, entry_tid=57031' 2026-02-22T19:54:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=55, tag_tid=2, entry_tid=57031' 2026-02-22T19:54:56.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=55, tag_tid=2, entry_tid=57031' '!=' 'object_number=54, tag_tid=2, entry_tid=56758' 2026-02-22T19:54:56.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:56.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=55, tag_tid=2, entry_tid=57031' = 'object_number=54, tag_tid=2, entry_tid=56758' 2026-02-22T19:54:56.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=55, tag_tid=2, entry_tid=57031' 2026-02-22T19:54:56.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:56.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:56.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:56.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=55, tag_tid=2, entry_tid=57031' 2026-02-22T19:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:56.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:56.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:56.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=55, tag_tid=2, entry_tid=57031' 2026-02-22T19:54:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:56.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=57031 2026-02-22T19:54:56.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 57031 2026-02-22T19:54:56.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:56.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:56.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:56.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:56.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:56.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:56.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:56.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:56.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:56.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:56.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:56.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:56.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:56.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:56.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:56.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:56.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:56.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:56.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:56.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:56.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=57324], [object_number=55, tag_tid=2, entry_tid=57323], [object_number=54, tag_tid=2, entry_tid=57322], [object_number=53, tag_tid=2, entry_tid=57321]]], state=connected] 2026-02-22T19:54:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:56.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:56.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=57324], [object_number=55, tag_tid=2, entry_tid=57323], [object_number=54, tag_tid=2, entry_tid=57322], [object_number=53, tag_tid=2, entry_tid=57321]]], state=connected] 2026-02-22T19:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=52, tag_tid=2, entry_tid=57324' 2026-02-22T19:54:56.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=52, tag_tid=2, entry_tid=57324' 2026-02-22T19:54:56.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=57324' '!=' 'object_number=55, tag_tid=2, entry_tid=57031' 2026-02-22T19:54:56.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:56.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=57324' = 'object_number=55, tag_tid=2, entry_tid=57031' 2026-02-22T19:54:56.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=52, tag_tid=2, entry_tid=57324' 2026-02-22T19:54:56.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:56.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=57324' 2026-02-22T19:54:56.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:56.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:56.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:56.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:56.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:56.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=57324' 2026-02-22T19:54:56.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:56.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=57324 2026-02-22T19:54:56.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 57324 2026-02-22T19:54:56.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:56.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:56.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:56.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:56.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:56.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:56.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:56.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:56.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:56.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:56.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:56.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:56.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:56.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:56.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:56.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:56.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:56.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=53, tag_tid=2, entry_tid=57585], [object_number=52, tag_tid=2, entry_tid=57584], [object_number=55, tag_tid=2, entry_tid=57583], [object_number=54, tag_tid=2, entry_tid=57582]]], state=connected] 2026-02-22T19:54:56.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:56.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:56.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:56.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:56.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:56.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:56.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:56.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=53, tag_tid=2, entry_tid=57585], [object_number=52, tag_tid=2, entry_tid=57584], [object_number=55, tag_tid=2, entry_tid=57583], [object_number=54, tag_tid=2, entry_tid=57582]]], state=connected] 2026-02-22T19:54:56.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:56.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=53, tag_tid=2, entry_tid=57585' 2026-02-22T19:54:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=53, tag_tid=2, entry_tid=57585' 2026-02-22T19:54:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=53, tag_tid=2, entry_tid=57585' '!=' 'object_number=52, tag_tid=2, entry_tid=57324' 2026-02-22T19:54:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=53, tag_tid=2, entry_tid=57585' = 'object_number=52, tag_tid=2, entry_tid=57324' 2026-02-22T19:54:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=53, tag_tid=2, entry_tid=57585' 2026-02-22T19:54:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:56.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:56.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:56.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=53, tag_tid=2, entry_tid=57585' 2026-02-22T19:54:56.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:56.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:56.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=53, tag_tid=2, entry_tid=57585' 2026-02-22T19:54:56.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:56.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:56.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=57585 2026-02-22T19:54:56.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 57585 2026-02-22T19:54:56.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:56.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:56.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:57.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:57.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:57.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:57.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:57.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:57.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:57.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:57.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:57.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:57.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:57.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:57.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:57.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:57.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:57.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:57.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:57.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:57.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:57.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:57.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=57920], [object_number=55, tag_tid=2, entry_tid=57919], [object_number=54, tag_tid=2, entry_tid=57918], [object_number=53, tag_tid=2, entry_tid=57917]]], state=connected] 2026-02-22T19:54:57.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:57.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:57.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=57920], [object_number=55, tag_tid=2, entry_tid=57919], [object_number=54, tag_tid=2, entry_tid=57918], [object_number=53, tag_tid=2, entry_tid=57917]]], state=connected] 2026-02-22T19:54:57.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=52, tag_tid=2, entry_tid=57920' 2026-02-22T19:54:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=52, tag_tid=2, entry_tid=57920' 2026-02-22T19:54:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=57920' '!=' 'object_number=53, tag_tid=2, entry_tid=57585' 2026-02-22T19:54:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=57920' = 'object_number=53, tag_tid=2, entry_tid=57585' 2026-02-22T19:54:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=52, tag_tid=2, entry_tid=57920' 2026-02-22T19:54:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:57.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:57.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:57.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=57920' 2026-02-22T19:54:57.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:57.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=57920' 2026-02-22T19:54:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:57.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=57920 2026-02-22T19:54:57.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 57920 2026-02-22T19:54:57.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:57.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:57.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:57.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:57.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:57.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:57.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:57.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:57.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:57.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:57.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:57.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:57.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:57.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:57.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:57.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:57.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=53, tag_tid=2, entry_tid=58205], [object_number=52, tag_tid=2, entry_tid=58204], [object_number=55, tag_tid=2, entry_tid=58203], [object_number=54, tag_tid=2, entry_tid=58202]]], state=connected] 2026-02-22T19:54:57.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:57.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:57.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:57.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:57.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:57.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:57.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:57.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:57.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:57.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:57.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:57.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:57.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=53, tag_tid=2, entry_tid=58205], [object_number=52, tag_tid=2, entry_tid=58204], [object_number=55, tag_tid=2, entry_tid=58203], [object_number=54, tag_tid=2, entry_tid=58202]]], state=connected] 2026-02-22T19:54:57.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=53, tag_tid=2, entry_tid=58205' 2026-02-22T19:54:57.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=53, tag_tid=2, entry_tid=58205' 2026-02-22T19:54:57.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=53, tag_tid=2, entry_tid=58205' '!=' 'object_number=52, tag_tid=2, entry_tid=57920' 2026-02-22T19:54:57.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:57.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=53, tag_tid=2, entry_tid=58205' = 'object_number=52, tag_tid=2, entry_tid=57920' 2026-02-22T19:54:57.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=53, tag_tid=2, entry_tid=58205' 2026-02-22T19:54:57.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:57.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:57.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:57.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:57.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=53, tag_tid=2, entry_tid=58205' 2026-02-22T19:54:57.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:57.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:57.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:57.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=53, tag_tid=2, entry_tid=58205' 2026-02-22T19:54:57.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:57.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=58205 2026-02-22T19:54:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 58205 2026-02-22T19:54:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:57.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:57.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:57.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:57.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:57.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:57.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:57.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:57.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:57.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:57.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:58.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:58.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:58.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:58.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=58472], [object_number=55, tag_tid=2, entry_tid=58471], [object_number=54, tag_tid=2, entry_tid=58470], [object_number=53, tag_tid=2, entry_tid=58469]]], state=connected] 2026-02-22T19:54:58.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 13 2026-02-22T19:54:58.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:58.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:58.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:58.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=52, tag_tid=2, entry_tid=58472], [object_number=55, tag_tid=2, entry_tid=58471], [object_number=54, tag_tid=2, entry_tid=58470], [object_number=53, tag_tid=2, entry_tid=58469]]], state=connected] 2026-02-22T19:54:58.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=52, tag_tid=2, entry_tid=58472' 2026-02-22T19:54:58.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=52, tag_tid=2, entry_tid=58472' 2026-02-22T19:54:58.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=58472' '!=' 'object_number=53, tag_tid=2, entry_tid=58205' 2026-02-22T19:54:58.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:58.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=52, tag_tid=2, entry_tid=58472' = 'object_number=53, tag_tid=2, entry_tid=58205' 2026-02-22T19:54:58.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=52, tag_tid=2, entry_tid=58472' 2026-02-22T19:54:58.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:58.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:58.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=58472' 2026-02-22T19:54:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:58.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:58.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:58.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:58.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=52, tag_tid=2, entry_tid=58472' 2026-02-22T19:54:58.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:58.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=58472 2026-02-22T19:54:58.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 58472 2026-02-22T19:54:58.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:58.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:58.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:58.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:58.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:58.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:58.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:58.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:58.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:58.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:58.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:58.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:58.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:58.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:58.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:58.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:58.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:58.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:58.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:58.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:58.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:58.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=58696], [object_number=59, tag_tid=2, entry_tid=58695], [object_number=58, tag_tid=2, entry_tid=58694], [object_number=57, tag_tid=2, entry_tid=58693]]], state=connected] 2026-02-22T19:54:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:58.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:58.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:58.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:58.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:58.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:58.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:58.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:58.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:58.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:58.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:58.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:58.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=58696], [object_number=59, tag_tid=2, entry_tid=58695], [object_number=58, tag_tid=2, entry_tid=58694], [object_number=57, tag_tid=2, entry_tid=58693]]], state=connected] 2026-02-22T19:54:58.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=56, tag_tid=2, entry_tid=58696' 2026-02-22T19:54:58.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=56, tag_tid=2, entry_tid=58696' 2026-02-22T19:54:58.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=58696' '!=' 'object_number=52, tag_tid=2, entry_tid=58472' 2026-02-22T19:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=58696' = 'object_number=52, tag_tid=2, entry_tid=58472' 2026-02-22T19:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=56, tag_tid=2, entry_tid=58696' 2026-02-22T19:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:58.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:58.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=58696' 2026-02-22T19:54:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:58.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:58.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:58.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:58.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=58696' 2026-02-22T19:54:58.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:58.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=58696 2026-02-22T19:54:58.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 58696 2026-02-22T19:54:58.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:58.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:58.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:58.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:58.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:58.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:58.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:58.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:58.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:58.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:58.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:58.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:58.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:58.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:58.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:58.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:58.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:58.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:58.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:58.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:58.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:58.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:58.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:58.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:58.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:58.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:58.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:58.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:58.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:58.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:58.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:58.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:58.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:58.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:58.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:58.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:58.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:58.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:58.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:58.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:58.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:58.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:58.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:58.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=57, tag_tid=2, entry_tid=58913], [object_number=56, tag_tid=2, entry_tid=58912], [object_number=59, tag_tid=2, entry_tid=58911], [object_number=58, tag_tid=2, entry_tid=58910]]], state=connected] 2026-02-22T19:54:58.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:58.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:58.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:58.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:58.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:58.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:58.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:58.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:58.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:58.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=57, tag_tid=2, entry_tid=58913], [object_number=56, tag_tid=2, entry_tid=58912], [object_number=59, tag_tid=2, entry_tid=58911], [object_number=58, tag_tid=2, entry_tid=58910]]], state=connected] 2026-02-22T19:54:58.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:58.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=57, tag_tid=2, entry_tid=58913' 2026-02-22T19:54:58.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=57, tag_tid=2, entry_tid=58913' 2026-02-22T19:54:58.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=57, tag_tid=2, entry_tid=58913' '!=' 'object_number=56, tag_tid=2, entry_tid=58696' 2026-02-22T19:54:58.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:58.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=57, tag_tid=2, entry_tid=58913' = 'object_number=56, tag_tid=2, entry_tid=58696' 2026-02-22T19:54:58.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=57, tag_tid=2, entry_tid=58913' 2026-02-22T19:54:58.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:58.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:58.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:58.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=57, tag_tid=2, entry_tid=58913' 2026-02-22T19:54:58.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:58.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:58.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:58.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:58.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=57, tag_tid=2, entry_tid=58913' 2026-02-22T19:54:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=58913 2026-02-22T19:54:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 58913 2026-02-22T19:54:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:59.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:59.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:59.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:59.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:59.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:59.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:59.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:59.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:59.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:59.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:59.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:59.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=57, tag_tid=2, entry_tid=59145], [object_number=56, tag_tid=2, entry_tid=59144], [object_number=59, tag_tid=2, entry_tid=59143], [object_number=58, tag_tid=2, entry_tid=59142]]], state=connected] 2026-02-22T19:54:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:59.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:59.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:59.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:59.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:59.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:59.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=57, tag_tid=2, entry_tid=59145], [object_number=56, tag_tid=2, entry_tid=59144], [object_number=59, tag_tid=2, entry_tid=59143], [object_number=58, tag_tid=2, entry_tid=59142]]], state=connected] 2026-02-22T19:54:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=57, tag_tid=2, entry_tid=59145' 2026-02-22T19:54:59.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=57, tag_tid=2, entry_tid=59145' 2026-02-22T19:54:59.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=57, tag_tid=2, entry_tid=59145' '!=' 'object_number=57, tag_tid=2, entry_tid=58913' 2026-02-22T19:54:59.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:59.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=57, tag_tid=2, entry_tid=59145' = 'object_number=57, tag_tid=2, entry_tid=58913' 2026-02-22T19:54:59.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=57, tag_tid=2, entry_tid=59145' 2026-02-22T19:54:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:59.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=57, tag_tid=2, entry_tid=59145' 2026-02-22T19:54:59.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:59.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:59.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:59.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:59.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=57, tag_tid=2, entry_tid=59145' 2026-02-22T19:54:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=59145 2026-02-22T19:54:59.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 59145 2026-02-22T19:54:59.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:59.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:59.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:59.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:59.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:59.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:59.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:59.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:59.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:59.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:59.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:59.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:59.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:59.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:59.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:59.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:59.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:59.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:59.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:59.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:59.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:59.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:59.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:59.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=59, tag_tid=2, entry_tid=59387], [object_number=58, tag_tid=2, entry_tid=59386], [object_number=57, tag_tid=2, entry_tid=59385], [object_number=56, tag_tid=2, entry_tid=59384]]], state=connected] 2026-02-22T19:54:59.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:59.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:59.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:59.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:59.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:59.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:59.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:59.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:59.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:59.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:59.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=59, tag_tid=2, entry_tid=59387], [object_number=58, tag_tid=2, entry_tid=59386], [object_number=57, tag_tid=2, entry_tid=59385], [object_number=56, tag_tid=2, entry_tid=59384]]], state=connected] 2026-02-22T19:54:59.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=59, tag_tid=2, entry_tid=59387' 2026-02-22T19:54:59.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=59, tag_tid=2, entry_tid=59387' 2026-02-22T19:54:59.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=59, tag_tid=2, entry_tid=59387' '!=' 'object_number=57, tag_tid=2, entry_tid=59145' 2026-02-22T19:54:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=59, tag_tid=2, entry_tid=59387' = 'object_number=57, tag_tid=2, entry_tid=59145' 2026-02-22T19:54:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=59, tag_tid=2, entry_tid=59387' 2026-02-22T19:54:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:59.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=59, tag_tid=2, entry_tid=59387' 2026-02-22T19:54:59.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:59.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:59.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:59.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:59.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:59.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=59, tag_tid=2, entry_tid=59387' 2026-02-22T19:54:59.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:59.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=59387 2026-02-22T19:54:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 59387 2026-02-22T19:54:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:59.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:59.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:54:59.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:54:59.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:54:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:59.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:54:59.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:59.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:59.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:54:59.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:54:59.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:59.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:54:59.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:54:59.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:54:59.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:54:59.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:54:59.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:59.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:59.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:59.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:59.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=58, tag_tid=2, entry_tid=59618], [object_number=57, tag_tid=2, entry_tid=59617], [object_number=56, tag_tid=2, entry_tid=59616], [object_number=59, tag_tid=2, entry_tid=59615]]], state=connected] 2026-02-22T19:54:59.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:54:59.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:54:59.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:54:59.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:54:59.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:54:59.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=58, tag_tid=2, entry_tid=59618], [object_number=57, tag_tid=2, entry_tid=59617], [object_number=56, tag_tid=2, entry_tid=59616], [object_number=59, tag_tid=2, entry_tid=59615]]], state=connected] 2026-02-22T19:54:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:54:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=58, tag_tid=2, entry_tid=59618' 2026-02-22T19:54:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=58, tag_tid=2, entry_tid=59618' 2026-02-22T19:54:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=58, tag_tid=2, entry_tid=59618' '!=' 'object_number=59, tag_tid=2, entry_tid=59387' 2026-02-22T19:54:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:54:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=58, tag_tid=2, entry_tid=59618' = 'object_number=59, tag_tid=2, entry_tid=59387' 2026-02-22T19:54:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=58, tag_tid=2, entry_tid=59618' 2026-02-22T19:54:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:54:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=58, tag_tid=2, entry_tid=59618' 2026-02-22T19:54:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:54:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:54:59.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:54:59.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:59.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:54:59.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=58, tag_tid=2, entry_tid=59618' 2026-02-22T19:54:59.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:54:59.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:54:59.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=59618 2026-02-22T19:54:59.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 59618 2026-02-22T19:54:59.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:54:59.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:59.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:00.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:00.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:00.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:00.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:00.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:00.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:00.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:00.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:00.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:00.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:00.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:00.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:00.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:00.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:00.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:00.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=59, tag_tid=2, entry_tid=59871], [object_number=58, tag_tid=2, entry_tid=59870], [object_number=57, tag_tid=2, entry_tid=59869], [object_number=56, tag_tid=2, entry_tid=59868]]], state=connected] 2026-02-22T19:55:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:00.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:00.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:00.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:00.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:00.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:00.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:00.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:00.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:00.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:00.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=59, tag_tid=2, entry_tid=59871], [object_number=58, tag_tid=2, entry_tid=59870], [object_number=57, tag_tid=2, entry_tid=59869], [object_number=56, tag_tid=2, entry_tid=59868]]], state=connected] 2026-02-22T19:55:00.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=59, tag_tid=2, entry_tid=59871' 2026-02-22T19:55:00.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=59, tag_tid=2, entry_tid=59871' 2026-02-22T19:55:00.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=59, tag_tid=2, entry_tid=59871' '!=' 'object_number=58, tag_tid=2, entry_tid=59618' 2026-02-22T19:55:00.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:00.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=59, tag_tid=2, entry_tid=59871' = 'object_number=58, tag_tid=2, entry_tid=59618' 2026-02-22T19:55:00.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=59, tag_tid=2, entry_tid=59871' 2026-02-22T19:55:00.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:00.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:00.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:00.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=59, tag_tid=2, entry_tid=59871' 2026-02-22T19:55:00.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:00.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:00.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:00.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:00.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:00.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=59, tag_tid=2, entry_tid=59871' 2026-02-22T19:55:00.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:00.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=59871 2026-02-22T19:55:00.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 59871 2026-02-22T19:55:00.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:00.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:00.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:00.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:00.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:00.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:00.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:00.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:00.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:00.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:00.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:00.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:00.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=58, tag_tid=2, entry_tid=60126], [object_number=57, tag_tid=2, entry_tid=60125], [object_number=56, tag_tid=2, entry_tid=60124], [object_number=59, tag_tid=2, entry_tid=60123]]], state=connected] 2026-02-22T19:55:00.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:00.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:00.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:00.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:00.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:00.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:00.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:00.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:00.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=58, tag_tid=2, entry_tid=60126], [object_number=57, tag_tid=2, entry_tid=60125], [object_number=56, tag_tid=2, entry_tid=60124], [object_number=59, tag_tid=2, entry_tid=60123]]], state=connected] 2026-02-22T19:55:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=58, tag_tid=2, entry_tid=60126' 2026-02-22T19:55:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=58, tag_tid=2, entry_tid=60126' 2026-02-22T19:55:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=58, tag_tid=2, entry_tid=60126' '!=' 'object_number=59, tag_tid=2, entry_tid=59871' 2026-02-22T19:55:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=58, tag_tid=2, entry_tid=60126' = 'object_number=59, tag_tid=2, entry_tid=59871' 2026-02-22T19:55:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=58, tag_tid=2, entry_tid=60126' 2026-02-22T19:55:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=58, tag_tid=2, entry_tid=60126' 2026-02-22T19:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:00.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:00.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:00.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:00.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=58, tag_tid=2, entry_tid=60126' 2026-02-22T19:55:00.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:00.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=60126 2026-02-22T19:55:00.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 60126 2026-02-22T19:55:00.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:00.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:00.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:00.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:00.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:00.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:00.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:00.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:00.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:00.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:00.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:00.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:00.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:00.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:00.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:00.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:00.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:00.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:00.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:00.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=60372], [object_number=59, tag_tid=2, entry_tid=60371], [object_number=58, tag_tid=2, entry_tid=60370], [object_number=57, tag_tid=2, entry_tid=60369]]], state=connected] 2026-02-22T19:55:00.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:00.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:00.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:00.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:00.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:00.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:00.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=60372], [object_number=59, tag_tid=2, entry_tid=60371], [object_number=58, tag_tid=2, entry_tid=60370], [object_number=57, tag_tid=2, entry_tid=60369]]], state=connected] 2026-02-22T19:55:00.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=56, tag_tid=2, entry_tid=60372' 2026-02-22T19:55:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=56, tag_tid=2, entry_tid=60372' 2026-02-22T19:55:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=60372' '!=' 'object_number=58, tag_tid=2, entry_tid=60126' 2026-02-22T19:55:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=60372' = 'object_number=58, tag_tid=2, entry_tid=60126' 2026-02-22T19:55:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=56, tag_tid=2, entry_tid=60372' 2026-02-22T19:55:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:00.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:00.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=60372' 2026-02-22T19:55:00.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:00.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:00.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:00.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:00.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=60372' 2026-02-22T19:55:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=60372 2026-02-22T19:55:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 60372 2026-02-22T19:55:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:01.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:01.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:01.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:01.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:01.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:01.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:01.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:01.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:01.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:01.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:01.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:01.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:01.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:01.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:01.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:01.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:01.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:01.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:01.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:01.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:01.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:01.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:01.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:01.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:01.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=60596], [object_number=59, tag_tid=2, entry_tid=60595], [object_number=58, tag_tid=2, entry_tid=60594], [object_number=57, tag_tid=2, entry_tid=60593]]], state=connected] 2026-02-22T19:55:01.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:01.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=60596], [object_number=59, tag_tid=2, entry_tid=60595], [object_number=58, tag_tid=2, entry_tid=60594], [object_number=57, tag_tid=2, entry_tid=60593]]], state=connected] 2026-02-22T19:55:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=56, tag_tid=2, entry_tid=60596' 2026-02-22T19:55:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=56, tag_tid=2, entry_tid=60596' 2026-02-22T19:55:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=60596' '!=' 'object_number=56, tag_tid=2, entry_tid=60372' 2026-02-22T19:55:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=60596' = 'object_number=56, tag_tid=2, entry_tid=60372' 2026-02-22T19:55:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=56, tag_tid=2, entry_tid=60596' 2026-02-22T19:55:01.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:01.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:01.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:01.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:01.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=60596' 2026-02-22T19:55:01.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:01.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:01.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:01.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:01.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:01.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:01.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:01.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=60596' 2026-02-22T19:55:01.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:01.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:01.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=60596 2026-02-22T19:55:01.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 60596 2026-02-22T19:55:01.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:01.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:01.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:01.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:01.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:01.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:01.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:01.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:01.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:01.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:01.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:01.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:01.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:01.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:01.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:01.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:01.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:01.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:01.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:01.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:01.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:01.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:01.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:01.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:01.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:01.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:01.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:01.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:01.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:01.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:01.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:01.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:01.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:01.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=59, tag_tid=2, entry_tid=60975], [object_number=58, tag_tid=2, entry_tid=60974], [object_number=57, tag_tid=2, entry_tid=60973], [object_number=56, tag_tid=2, entry_tid=60972]]], state=connected] 2026-02-22T19:55:01.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:01.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:01.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:01.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:01.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:01.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:01.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:01.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:01.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:01.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:01.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:02.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:02.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:02.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:02.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=59, tag_tid=2, entry_tid=60975], [object_number=58, tag_tid=2, entry_tid=60974], [object_number=57, tag_tid=2, entry_tid=60973], [object_number=56, tag_tid=2, entry_tid=60972]]], state=connected] 2026-02-22T19:55:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=59, tag_tid=2, entry_tid=60975' 2026-02-22T19:55:02.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=59, tag_tid=2, entry_tid=60975' 2026-02-22T19:55:02.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=59, tag_tid=2, entry_tid=60975' '!=' 'object_number=56, tag_tid=2, entry_tid=60596' 2026-02-22T19:55:02.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=59, tag_tid=2, entry_tid=60975' = 'object_number=56, tag_tid=2, entry_tid=60596' 2026-02-22T19:55:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=59, tag_tid=2, entry_tid=60975' 2026-02-22T19:55:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=59, tag_tid=2, entry_tid=60975' 2026-02-22T19:55:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=59, tag_tid=2, entry_tid=60975' 2026-02-22T19:55:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=60975 2026-02-22T19:55:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 60975 2026-02-22T19:55:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:02.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:02.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:02.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:02.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:02.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:02.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:02.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:02.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:02.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:02.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:02.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:02.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:02.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:02.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:02.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=61388], [object_number=59, tag_tid=2, entry_tid=61387], [object_number=58, tag_tid=2, entry_tid=61386], [object_number=57, tag_tid=2, entry_tid=61385]]], state=connected] 2026-02-22T19:55:02.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:02.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:02.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:02.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:02.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:02.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:02.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:02.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:02.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:02.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:02.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:02.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:02.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:02.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:02.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=61388], [object_number=59, tag_tid=2, entry_tid=61387], [object_number=58, tag_tid=2, entry_tid=61386], [object_number=57, tag_tid=2, entry_tid=61385]]], state=connected] 2026-02-22T19:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=56, tag_tid=2, entry_tid=61388' 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=56, tag_tid=2, entry_tid=61388' 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=61388' '!=' 'object_number=59, tag_tid=2, entry_tid=60975' 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=61388' = 'object_number=59, tag_tid=2, entry_tid=60975' 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=56, tag_tid=2, entry_tid=61388' 2026-02-22T19:55:02.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:02.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:02.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=61388' 2026-02-22T19:55:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:02.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:02.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:02.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=61388' 2026-02-22T19:55:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=61388 2026-02-22T19:55:02.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 61388 2026-02-22T19:55:02.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:02.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:02.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:02.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:02.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:02.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:02.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:02.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:02.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:02.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:02.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:02.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:02.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:02.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:02.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:02.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:02.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:02.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:02.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:02.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:02.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:02.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:02.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:02.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:02.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=58, tag_tid=2, entry_tid=61678], [object_number=57, tag_tid=2, entry_tid=61677], [object_number=56, tag_tid=2, entry_tid=61676], [object_number=59, tag_tid=2, entry_tid=61675]]], state=connected] 2026-02-22T19:55:02.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:02.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:02.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=58, tag_tid=2, entry_tid=61678], [object_number=57, tag_tid=2, entry_tid=61677], [object_number=56, tag_tid=2, entry_tid=61676], [object_number=59, tag_tid=2, entry_tid=61675]]], state=connected] 2026-02-22T19:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=58, tag_tid=2, entry_tid=61678' 2026-02-22T19:55:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=58, tag_tid=2, entry_tid=61678' 2026-02-22T19:55:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=58, tag_tid=2, entry_tid=61678' '!=' 'object_number=56, tag_tid=2, entry_tid=61388' 2026-02-22T19:55:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=58, tag_tid=2, entry_tid=61678' = 'object_number=56, tag_tid=2, entry_tid=61388' 2026-02-22T19:55:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=58, tag_tid=2, entry_tid=61678' 2026-02-22T19:55:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:02.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:02.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:02.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=58, tag_tid=2, entry_tid=61678' 2026-02-22T19:55:02.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:02.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:02.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=58, tag_tid=2, entry_tid=61678' 2026-02-22T19:55:02.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:02.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:02.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=61678 2026-02-22T19:55:02.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 61678 2026-02-22T19:55:02.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:02.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:02.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:03.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:03.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:03.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:03.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:03.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:03.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:03.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=61956], [object_number=59, tag_tid=2, entry_tid=61955], [object_number=58, tag_tid=2, entry_tid=61954], [object_number=57, tag_tid=2, entry_tid=61953]]], state=connected] 2026-02-22T19:55:03.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:03.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:03.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:03.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:03.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:03.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:03.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=56, tag_tid=2, entry_tid=61956], [object_number=59, tag_tid=2, entry_tid=61955], [object_number=58, tag_tid=2, entry_tid=61954], [object_number=57, tag_tid=2, entry_tid=61953]]], state=connected] 2026-02-22T19:55:03.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=56, tag_tid=2, entry_tid=61956' 2026-02-22T19:55:03.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=56, tag_tid=2, entry_tid=61956' 2026-02-22T19:55:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=61956' '!=' 'object_number=58, tag_tid=2, entry_tid=61678' 2026-02-22T19:55:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=56, tag_tid=2, entry_tid=61956' = 'object_number=58, tag_tid=2, entry_tid=61678' 2026-02-22T19:55:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=56, tag_tid=2, entry_tid=61956' 2026-02-22T19:55:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:03.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:03.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:03.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=61956' 2026-02-22T19:55:03.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:03.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:03.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:03.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=56, tag_tid=2, entry_tid=61956' 2026-02-22T19:55:03.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:03.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=61956 2026-02-22T19:55:03.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 61956 2026-02-22T19:55:03.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:03.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:03.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:03.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:03.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:03.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:03.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:03.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:03.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:03.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:03.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:03.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:03.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:03.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:03.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:03.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:03.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:03.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:03.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:03.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:03.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:03.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:03.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:03.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=57, tag_tid=2, entry_tid=62201], [object_number=56, tag_tid=2, entry_tid=62200], [object_number=59, tag_tid=2, entry_tid=62199], [object_number=58, tag_tid=2, entry_tid=62198]]], state=connected] 2026-02-22T19:55:03.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:03.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:03.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:03.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:03.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:03.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:03.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:03.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:03.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:03.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:03.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=57, tag_tid=2, entry_tid=62201], [object_number=56, tag_tid=2, entry_tid=62200], [object_number=59, tag_tid=2, entry_tid=62199], [object_number=58, tag_tid=2, entry_tid=62198]]], state=connected] 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=57, tag_tid=2, entry_tid=62201' 2026-02-22T19:55:03.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=57, tag_tid=2, entry_tid=62201' 2026-02-22T19:55:03.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=57, tag_tid=2, entry_tid=62201' '!=' 'object_number=56, tag_tid=2, entry_tid=61956' 2026-02-22T19:55:03.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:03.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=57, tag_tid=2, entry_tid=62201' = 'object_number=56, tag_tid=2, entry_tid=61956' 2026-02-22T19:55:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=57, tag_tid=2, entry_tid=62201' 2026-02-22T19:55:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=57, tag_tid=2, entry_tid=62201' 2026-02-22T19:55:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:03.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:03.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:03.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:03.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=57, tag_tid=2, entry_tid=62201' 2026-02-22T19:55:03.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:03.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=62201 2026-02-22T19:55:03.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 62201 2026-02-22T19:55:03.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:03.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:03.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:03.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:03.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:03.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:03.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:03.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:03.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:03.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:03.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:03.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:03.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:03.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:03.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:03.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:03.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=57, tag_tid=2, entry_tid=62461], [object_number=56, tag_tid=2, entry_tid=62460], [object_number=59, tag_tid=2, entry_tid=62459], [object_number=58, tag_tid=2, entry_tid=62458]]], state=connected] 2026-02-22T19:55:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:03.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:03.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:03.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:03.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:03.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=57, tag_tid=2, entry_tid=62461], [object_number=56, tag_tid=2, entry_tid=62460], [object_number=59, tag_tid=2, entry_tid=62459], [object_number=58, tag_tid=2, entry_tid=62458]]], state=connected] 2026-02-22T19:55:03.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=57, tag_tid=2, entry_tid=62461' 2026-02-22T19:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=57, tag_tid=2, entry_tid=62461' 2026-02-22T19:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=57, tag_tid=2, entry_tid=62461' '!=' 'object_number=57, tag_tid=2, entry_tid=62201' 2026-02-22T19:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=57, tag_tid=2, entry_tid=62461' = 'object_number=57, tag_tid=2, entry_tid=62201' 2026-02-22T19:55:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=57, tag_tid=2, entry_tid=62461' 2026-02-22T19:55:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:03.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=57, tag_tid=2, entry_tid=62461' 2026-02-22T19:55:03.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:03.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:03.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:03.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:03.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:03.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=57, tag_tid=2, entry_tid=62461' 2026-02-22T19:55:03.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:03.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:03.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=62461 2026-02-22T19:55:03.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 62461 2026-02-22T19:55:03.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:03.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:03.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:04.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:04.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:04.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:04.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:04.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:04.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:04.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=62641], [object_number=60, tag_tid=2, entry_tid=62640], [object_number=59, tag_tid=2, entry_tid=62639], [object_number=58, tag_tid=2, entry_tid=62638]]], state=connected] 2026-02-22T19:55:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 14 2026-02-22T19:55:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=62641], [object_number=60, tag_tid=2, entry_tid=62640], [object_number=59, tag_tid=2, entry_tid=62639], [object_number=58, tag_tid=2, entry_tid=62638]]], state=connected] 2026-02-22T19:55:04.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=61, tag_tid=2, entry_tid=62641' 2026-02-22T19:55:04.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=61, tag_tid=2, entry_tid=62641' 2026-02-22T19:55:04.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=62641' '!=' 'object_number=57, tag_tid=2, entry_tid=62461' 2026-02-22T19:55:04.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:04.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=62641' = 'object_number=57, tag_tid=2, entry_tid=62461' 2026-02-22T19:55:04.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=61, tag_tid=2, entry_tid=62641' 2026-02-22T19:55:04.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:04.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:04.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:04.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=62641' 2026-02-22T19:55:04.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:04.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:04.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:04.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:04.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:04.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=62641' 2026-02-22T19:55:04.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:04.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=62641 2026-02-22T19:55:04.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 62641 2026-02-22T19:55:04.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:04.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:04.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:04.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:04.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:04.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:04.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:04.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:04.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:04.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:04.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:04.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:04.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:04.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:04.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:04.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:04.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:04.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:04.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:04.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:04.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:04.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:04.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:04.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:04.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:04.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:04.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:04.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:04.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=63, tag_tid=2, entry_tid=62863], [object_number=62, tag_tid=2, entry_tid=62862], [object_number=61, tag_tid=2, entry_tid=62861], [object_number=60, tag_tid=2, entry_tid=62860]]], state=connected] 2026-02-22T19:55:04.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:04.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:04.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:04.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:04.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:04.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:04.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:04.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:04.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=63, tag_tid=2, entry_tid=62863], [object_number=62, tag_tid=2, entry_tid=62862], [object_number=61, tag_tid=2, entry_tid=62861], [object_number=60, tag_tid=2, entry_tid=62860]]], state=connected] 2026-02-22T19:55:04.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=63, tag_tid=2, entry_tid=62863' 2026-02-22T19:55:04.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=63, tag_tid=2, entry_tid=62863' 2026-02-22T19:55:04.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=63, tag_tid=2, entry_tid=62863' '!=' 'object_number=61, tag_tid=2, entry_tid=62641' 2026-02-22T19:55:04.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:04.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=63, tag_tid=2, entry_tid=62863' = 'object_number=61, tag_tid=2, entry_tid=62641' 2026-02-22T19:55:04.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=63, tag_tid=2, entry_tid=62863' 2026-02-22T19:55:04.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:04.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:04.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:04.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=63, tag_tid=2, entry_tid=62863' 2026-02-22T19:55:04.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:04.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:04.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:04.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:04.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:04.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=63, tag_tid=2, entry_tid=62863' 2026-02-22T19:55:04.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:04.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=62863 2026-02-22T19:55:04.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 62863 2026-02-22T19:55:04.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:04.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:04.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:04.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:04.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:04.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:04.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:04.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:04.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:04.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:04.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:04.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:04.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:04.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:04.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:04.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:04.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:04.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:04.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:04.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:04.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:04.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:04.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:04.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:04.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:04.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:04.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:04.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:04.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:04.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:04.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:04.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:04.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:04.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:04.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:04.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=62, tag_tid=2, entry_tid=63142], [object_number=61, tag_tid=2, entry_tid=63141], [object_number=60, tag_tid=2, entry_tid=63140], [object_number=63, tag_tid=2, entry_tid=63139]]], state=connected] 2026-02-22T19:55:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:04.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:04.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:04.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:04.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:04.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:04.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:04.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:04.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:04.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:04.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:04.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=62, tag_tid=2, entry_tid=63142], [object_number=61, tag_tid=2, entry_tid=63141], [object_number=60, tag_tid=2, entry_tid=63140], [object_number=63, tag_tid=2, entry_tid=63139]]], state=connected] 2026-02-22T19:55:04.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:04.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=62, tag_tid=2, entry_tid=63142' 2026-02-22T19:55:04.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=62, tag_tid=2, entry_tid=63142' 2026-02-22T19:55:04.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=62, tag_tid=2, entry_tid=63142' '!=' 'object_number=63, tag_tid=2, entry_tid=62863' 2026-02-22T19:55:04.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:04.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=62, tag_tid=2, entry_tid=63142' = 'object_number=63, tag_tid=2, entry_tid=62863' 2026-02-22T19:55:04.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=62, tag_tid=2, entry_tid=63142' 2026-02-22T19:55:04.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:04.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:04.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:04.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=62, tag_tid=2, entry_tid=63142' 2026-02-22T19:55:04.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:04.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=62, tag_tid=2, entry_tid=63142' 2026-02-22T19:55:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=63142 2026-02-22T19:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 63142 2026-02-22T19:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:05.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:05.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:05.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:05.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:05.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=63409], [object_number=60, tag_tid=2, entry_tid=63408], [object_number=63, tag_tid=2, entry_tid=63407], [object_number=62, tag_tid=2, entry_tid=63406]]], state=connected] 2026-02-22T19:55:05.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:05.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:05.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:05.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:05.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=63409], [object_number=60, tag_tid=2, entry_tid=63408], [object_number=63, tag_tid=2, entry_tid=63407], [object_number=62, tag_tid=2, entry_tid=63406]]], state=connected] 2026-02-22T19:55:05.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=61, tag_tid=2, entry_tid=63409' 2026-02-22T19:55:05.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=61, tag_tid=2, entry_tid=63409' 2026-02-22T19:55:05.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=63409' '!=' 'object_number=62, tag_tid=2, entry_tid=63142' 2026-02-22T19:55:05.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:05.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=63409' = 'object_number=62, tag_tid=2, entry_tid=63142' 2026-02-22T19:55:05.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=61, tag_tid=2, entry_tid=63409' 2026-02-22T19:55:05.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:05.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:05.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:05.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:05.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=63409' 2026-02-22T19:55:05.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:05.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:05.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:05.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:05.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:05.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:05.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:05.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=63409' 2026-02-22T19:55:05.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:05.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=63409 2026-02-22T19:55:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 63409 2026-02-22T19:55:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:05.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:05.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:05.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:05.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:05.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:05.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:05.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:05.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:05.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:05.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=63653], [object_number=60, tag_tid=2, entry_tid=63652], [object_number=63, tag_tid=2, entry_tid=63651], [object_number=62, tag_tid=2, entry_tid=63650]]], state=connected] 2026-02-22T19:55:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:05.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:05.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:05.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:05.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=63653], [object_number=60, tag_tid=2, entry_tid=63652], [object_number=63, tag_tid=2, entry_tid=63651], [object_number=62, tag_tid=2, entry_tid=63650]]], state=connected] 2026-02-22T19:55:05.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=61, tag_tid=2, entry_tid=63653' 2026-02-22T19:55:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=61, tag_tid=2, entry_tid=63653' 2026-02-22T19:55:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=63653' '!=' 'object_number=61, tag_tid=2, entry_tid=63409' 2026-02-22T19:55:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=63653' = 'object_number=61, tag_tid=2, entry_tid=63409' 2026-02-22T19:55:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=61, tag_tid=2, entry_tid=63653' 2026-02-22T19:55:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=63653' 2026-02-22T19:55:05.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:05.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:05.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:05.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:05.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:05.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:05.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:05.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=63653' 2026-02-22T19:55:05.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:05.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=63653 2026-02-22T19:55:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 63653 2026-02-22T19:55:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:05.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:05.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:05.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:05.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:05.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:05.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:05.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:05.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:05.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:05.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:05.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:05.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:05.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:05.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:05.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:05.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=62, tag_tid=2, entry_tid=63898], [object_number=61, tag_tid=2, entry_tid=63897], [object_number=60, tag_tid=2, entry_tid=63896], [object_number=63, tag_tid=2, entry_tid=63895]]], state=connected] 2026-02-22T19:55:05.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:06.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:06.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:06.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:06.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:06.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:06.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:06.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:06.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:06.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=62, tag_tid=2, entry_tid=63898], [object_number=61, tag_tid=2, entry_tid=63897], [object_number=60, tag_tid=2, entry_tid=63896], [object_number=63, tag_tid=2, entry_tid=63895]]], state=connected] 2026-02-22T19:55:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=62, tag_tid=2, entry_tid=63898' 2026-02-22T19:55:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=62, tag_tid=2, entry_tid=63898' 2026-02-22T19:55:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=62, tag_tid=2, entry_tid=63898' '!=' 'object_number=61, tag_tid=2, entry_tid=63653' 2026-02-22T19:55:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=62, tag_tid=2, entry_tid=63898' = 'object_number=61, tag_tid=2, entry_tid=63653' 2026-02-22T19:55:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=62, tag_tid=2, entry_tid=63898' 2026-02-22T19:55:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=62, tag_tid=2, entry_tid=63898' 2026-02-22T19:55:06.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:06.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:06.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:06.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:06.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:06.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=62, tag_tid=2, entry_tid=63898' 2026-02-22T19:55:06.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:06.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=63898 2026-02-22T19:55:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 63898 2026-02-22T19:55:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:06.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:06.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:06.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:06.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:06.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:06.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:06.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:06.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:06.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:06.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:06.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:06.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=64144], [object_number=63, tag_tid=2, entry_tid=64143], [object_number=62, tag_tid=2, entry_tid=64142], [object_number=61, tag_tid=2, entry_tid=64141]]], state=connected] 2026-02-22T19:55:06.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:06.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:06.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=64144], [object_number=63, tag_tid=2, entry_tid=64143], [object_number=62, tag_tid=2, entry_tid=64142], [object_number=61, tag_tid=2, entry_tid=64141]]], state=connected] 2026-02-22T19:55:06.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=60, tag_tid=2, entry_tid=64144' 2026-02-22T19:55:06.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=60, tag_tid=2, entry_tid=64144' 2026-02-22T19:55:06.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=64144' '!=' 'object_number=62, tag_tid=2, entry_tid=63898' 2026-02-22T19:55:06.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:06.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=64144' = 'object_number=62, tag_tid=2, entry_tid=63898' 2026-02-22T19:55:06.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=60, tag_tid=2, entry_tid=64144' 2026-02-22T19:55:06.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:06.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:06.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:06.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=64144' 2026-02-22T19:55:06.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:06.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:06.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:06.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:06.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:06.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=64144' 2026-02-22T19:55:06.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:06.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=64144 2026-02-22T19:55:06.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 64144 2026-02-22T19:55:06.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:06.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:06.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:06.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:06.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:06.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:06.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:06.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:06.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:06.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:06.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:06.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:06.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:06.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:06.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:06.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:06.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:06.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:06.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:06.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:06.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:06.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:06.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=62, tag_tid=2, entry_tid=64378], [object_number=61, tag_tid=2, entry_tid=64377], [object_number=60, tag_tid=2, entry_tid=64376], [object_number=63, tag_tid=2, entry_tid=64375]]], state=connected] 2026-02-22T19:55:06.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:06.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:06.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:06.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:06.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:06.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:06.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:06.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:06.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:06.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:06.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=62, tag_tid=2, entry_tid=64378], [object_number=61, tag_tid=2, entry_tid=64377], [object_number=60, tag_tid=2, entry_tid=64376], [object_number=63, tag_tid=2, entry_tid=64375]]], state=connected] 2026-02-22T19:55:06.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=62, tag_tid=2, entry_tid=64378' 2026-02-22T19:55:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=62, tag_tid=2, entry_tid=64378' 2026-02-22T19:55:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=62, tag_tid=2, entry_tid=64378' '!=' 'object_number=60, tag_tid=2, entry_tid=64144' 2026-02-22T19:55:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=62, tag_tid=2, entry_tid=64378' = 'object_number=60, tag_tid=2, entry_tid=64144' 2026-02-22T19:55:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=62, tag_tid=2, entry_tid=64378' 2026-02-22T19:55:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:06.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=62, tag_tid=2, entry_tid=64378' 2026-02-22T19:55:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:06.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:06.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:06.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:06.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=62, tag_tid=2, entry_tid=64378' 2026-02-22T19:55:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:06.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=64378 2026-02-22T19:55:06.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 64378 2026-02-22T19:55:06.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:06.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:06.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:06.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:06.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:06.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:06.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:06.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:06.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:06.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:06.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:06.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:06.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:06.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:06.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=64624], [object_number=63, tag_tid=2, entry_tid=64623], [object_number=62, tag_tid=2, entry_tid=64622], [object_number=61, tag_tid=2, entry_tid=64621]]], state=connected] 2026-02-22T19:55:07.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:07.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:07.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:07.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:07.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:07.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:07.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:07.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:07.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:07.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=64624], [object_number=63, tag_tid=2, entry_tid=64623], [object_number=62, tag_tid=2, entry_tid=64622], [object_number=61, tag_tid=2, entry_tid=64621]]], state=connected] 2026-02-22T19:55:07.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=60, tag_tid=2, entry_tid=64624' 2026-02-22T19:55:07.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=60, tag_tid=2, entry_tid=64624' 2026-02-22T19:55:07.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=64624' '!=' 'object_number=62, tag_tid=2, entry_tid=64378' 2026-02-22T19:55:07.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:07.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=64624' = 'object_number=62, tag_tid=2, entry_tid=64378' 2026-02-22T19:55:07.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=60, tag_tid=2, entry_tid=64624' 2026-02-22T19:55:07.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:07.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:07.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:07.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=64624' 2026-02-22T19:55:07.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:07.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:07.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:07.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:07.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:07.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=64624' 2026-02-22T19:55:07.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:07.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=64624 2026-02-22T19:55:07.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 64624 2026-02-22T19:55:07.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:07.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:07.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:07.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:07.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:07.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:07.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:07.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:07.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:07.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:07.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:07.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:07.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:07.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:07.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:07.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:07.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:07.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:07.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=64905], [object_number=60, tag_tid=2, entry_tid=64904], [object_number=63, tag_tid=2, entry_tid=64903], [object_number=62, tag_tid=2, entry_tid=64902]]], state=connected] 2026-02-22T19:55:07.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:07.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:07.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:07.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=64905], [object_number=60, tag_tid=2, entry_tid=64904], [object_number=63, tag_tid=2, entry_tid=64903], [object_number=62, tag_tid=2, entry_tid=64902]]], state=connected] 2026-02-22T19:55:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=61, tag_tid=2, entry_tid=64905' 2026-02-22T19:55:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=61, tag_tid=2, entry_tid=64905' 2026-02-22T19:55:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=64905' '!=' 'object_number=60, tag_tid=2, entry_tid=64624' 2026-02-22T19:55:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=64905' = 'object_number=60, tag_tid=2, entry_tid=64624' 2026-02-22T19:55:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=61, tag_tid=2, entry_tid=64905' 2026-02-22T19:55:07.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:07.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:07.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=64905' 2026-02-22T19:55:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:07.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:07.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:07.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:07.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=64905' 2026-02-22T19:55:07.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:07.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=64905 2026-02-22T19:55:07.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 64905 2026-02-22T19:55:07.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:07.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:07.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:07.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:07.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:07.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:07.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:07.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:07.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:07.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:07.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:07.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:07.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:07.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:07.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:07.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:07.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:07.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:07.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:07.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:07.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=65164], [object_number=63, tag_tid=2, entry_tid=65163], [object_number=62, tag_tid=2, entry_tid=65162], [object_number=61, tag_tid=2, entry_tid=65161]]], state=connected] 2026-02-22T19:55:07.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:07.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:07.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:07.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:07.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:07.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:07.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:07.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:07.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=65164], [object_number=63, tag_tid=2, entry_tid=65163], [object_number=62, tag_tid=2, entry_tid=65162], [object_number=61, tag_tid=2, entry_tid=65161]]], state=connected] 2026-02-22T19:55:07.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:07.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=60, tag_tid=2, entry_tid=65164' 2026-02-22T19:55:07.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=60, tag_tid=2, entry_tid=65164' 2026-02-22T19:55:07.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=65164' '!=' 'object_number=61, tag_tid=2, entry_tid=64905' 2026-02-22T19:55:07.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:07.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=65164' = 'object_number=61, tag_tid=2, entry_tid=64905' 2026-02-22T19:55:07.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=60, tag_tid=2, entry_tid=65164' 2026-02-22T19:55:07.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:07.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:07.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:07.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=65164' 2026-02-22T19:55:07.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:07.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:07.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:07.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:07.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=65164' 2026-02-22T19:55:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=65164 2026-02-22T19:55:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 65164 2026-02-22T19:55:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:08.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:08.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:08.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:08.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:08.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:08.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:08.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:08.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:08.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:08.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:08.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:08.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:08.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:08.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:08.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:08.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:08.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:08.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:08.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:08.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:08.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:08.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:08.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:08.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:08.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:08.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:08.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:08.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:08.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:08.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:08.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:08.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=65428], [object_number=63, tag_tid=2, entry_tid=65427], [object_number=62, tag_tid=2, entry_tid=65426], [object_number=61, tag_tid=2, entry_tid=65425]]], state=connected] 2026-02-22T19:55:08.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:08.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:08.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:08.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:08.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:08.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:08.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=65428], [object_number=63, tag_tid=2, entry_tid=65427], [object_number=62, tag_tid=2, entry_tid=65426], [object_number=61, tag_tid=2, entry_tid=65425]]], state=connected] 2026-02-22T19:55:08.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=60, tag_tid=2, entry_tid=65428' 2026-02-22T19:55:08.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=60, tag_tid=2, entry_tid=65428' 2026-02-22T19:55:08.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=65428' '!=' 'object_number=60, tag_tid=2, entry_tid=65164' 2026-02-22T19:55:08.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:08.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=65428' = 'object_number=60, tag_tid=2, entry_tid=65164' 2026-02-22T19:55:08.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=60, tag_tid=2, entry_tid=65428' 2026-02-22T19:55:08.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:08.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:08.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=65428' 2026-02-22T19:55:08.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:08.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:08.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:08.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:08.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:08.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:08.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:08.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=65428' 2026-02-22T19:55:08.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:08.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:08.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=65428 2026-02-22T19:55:08.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 65428 2026-02-22T19:55:08.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:08.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:08.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:08.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:08.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:08.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:08.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:08.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:08.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:08.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:08.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:08.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:08.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:08.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:08.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:08.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:08.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:08.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:08.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:08.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:08.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:08.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:08.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:08.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:08.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:08.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:08.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:08.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:08.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:08.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:08.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:08.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:08.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:08.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:08.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:08.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=65700], [object_number=63, tag_tid=2, entry_tid=65699], [object_number=62, tag_tid=2, entry_tid=65698], [object_number=61, tag_tid=2, entry_tid=65697]]], state=connected] 2026-02-22T19:55:08.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:08.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:08.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:08.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:08.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=65700], [object_number=63, tag_tid=2, entry_tid=65699], [object_number=62, tag_tid=2, entry_tid=65698], [object_number=61, tag_tid=2, entry_tid=65697]]], state=connected] 2026-02-22T19:55:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=60, tag_tid=2, entry_tid=65700' 2026-02-22T19:55:08.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=60, tag_tid=2, entry_tid=65700' 2026-02-22T19:55:08.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=65700' '!=' 'object_number=60, tag_tid=2, entry_tid=65428' 2026-02-22T19:55:08.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:08.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=65700' = 'object_number=60, tag_tid=2, entry_tid=65428' 2026-02-22T19:55:08.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=60, tag_tid=2, entry_tid=65700' 2026-02-22T19:55:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=65700' 2026-02-22T19:55:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:08.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:08.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=65700' 2026-02-22T19:55:08.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:08.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=65700 2026-02-22T19:55:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 65700 2026-02-22T19:55:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:08.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:08.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:08.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:08.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:08.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:08.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:08.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:08.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:08.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=65937], [object_number=60, tag_tid=2, entry_tid=65936], [object_number=63, tag_tid=2, entry_tid=65935], [object_number=62, tag_tid=2, entry_tid=65934]]], state=connected] 2026-02-22T19:55:08.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:08.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:08.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:08.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:08.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:08.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:08.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:08.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:09.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:09.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:09.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=65937], [object_number=60, tag_tid=2, entry_tid=65936], [object_number=63, tag_tid=2, entry_tid=65935], [object_number=62, tag_tid=2, entry_tid=65934]]], state=connected] 2026-02-22T19:55:09.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=61, tag_tid=2, entry_tid=65937' 2026-02-22T19:55:09.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=61, tag_tid=2, entry_tid=65937' 2026-02-22T19:55:09.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=65937' '!=' 'object_number=60, tag_tid=2, entry_tid=65700' 2026-02-22T19:55:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=65937' = 'object_number=60, tag_tid=2, entry_tid=65700' 2026-02-22T19:55:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=61, tag_tid=2, entry_tid=65937' 2026-02-22T19:55:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=65937' 2026-02-22T19:55:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:09.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:09.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:09.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:09.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=65937' 2026-02-22T19:55:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=65937 2026-02-22T19:55:09.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 65937 2026-02-22T19:55:09.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:09.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:09.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:09.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:09.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:09.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:09.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:09.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:09.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:09.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:09.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:09.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:09.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:09.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:09.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:09.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:09.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:09.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:09.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:09.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:09.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:09.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:09.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:09.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:09.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:09.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:09.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:09.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:09.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=66293], [object_number=60, tag_tid=2, entry_tid=66292], [object_number=63, tag_tid=2, entry_tid=66291], [object_number=62, tag_tid=2, entry_tid=66290]]], state=connected] 2026-02-22T19:55:09.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:09.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:09.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:09.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:09.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:09.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:09.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:09.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:09.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=61, tag_tid=2, entry_tid=66293], [object_number=60, tag_tid=2, entry_tid=66292], [object_number=63, tag_tid=2, entry_tid=66291], [object_number=62, tag_tid=2, entry_tid=66290]]], state=connected] 2026-02-22T19:55:09.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=61, tag_tid=2, entry_tid=66293' 2026-02-22T19:55:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=61, tag_tid=2, entry_tid=66293' 2026-02-22T19:55:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=66293' '!=' 'object_number=61, tag_tid=2, entry_tid=65937' 2026-02-22T19:55:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=61, tag_tid=2, entry_tid=66293' = 'object_number=61, tag_tid=2, entry_tid=65937' 2026-02-22T19:55:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=61, tag_tid=2, entry_tid=66293' 2026-02-22T19:55:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:09.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=66293' 2026-02-22T19:55:09.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:09.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=61, tag_tid=2, entry_tid=66293' 2026-02-22T19:55:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=66293 2026-02-22T19:55:09.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 66293 2026-02-22T19:55:09.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:09.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:09.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:09.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:09.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:09.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:09.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:09.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:09.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:09.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:09.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:09.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:09.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:09.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:09.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=66500], [object_number=63, tag_tid=2, entry_tid=66499], [object_number=62, tag_tid=2, entry_tid=66498], [object_number=61, tag_tid=2, entry_tid=66497]]], state=connected] 2026-02-22T19:55:09.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:09.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:09.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:09.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=60, tag_tid=2, entry_tid=66500], [object_number=63, tag_tid=2, entry_tid=66499], [object_number=62, tag_tid=2, entry_tid=66498], [object_number=61, tag_tid=2, entry_tid=66497]]], state=connected] 2026-02-22T19:55:09.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:09.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=60, tag_tid=2, entry_tid=66500' 2026-02-22T19:55:09.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=60, tag_tid=2, entry_tid=66500' 2026-02-22T19:55:09.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=66500' '!=' 'object_number=61, tag_tid=2, entry_tid=66293' 2026-02-22T19:55:09.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:09.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=60, tag_tid=2, entry_tid=66500' = 'object_number=61, tag_tid=2, entry_tid=66293' 2026-02-22T19:55:09.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=60, tag_tid=2, entry_tid=66500' 2026-02-22T19:55:09.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:09.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:09.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:09.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=66500' 2026-02-22T19:55:09.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:09.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:09.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:09.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:09.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:09.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=60, tag_tid=2, entry_tid=66500' 2026-02-22T19:55:09.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:09.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:09.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=66500 2026-02-22T19:55:09.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 66500 2026-02-22T19:55:09.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:09.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:09.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:10.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:10.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:10.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:10.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:10.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:10.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:10.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:10.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:10.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:10.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:10.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:10.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=62, tag_tid=2, entry_tid=66742], [object_number=61, tag_tid=2, entry_tid=66741], [object_number=60, tag_tid=2, entry_tid=66740], [object_number=63, tag_tid=2, entry_tid=66739]]], state=connected] 2026-02-22T19:55:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:10.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:10.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:10.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:10.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:10.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:10.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:10.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:10.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 15 2026-02-22T19:55:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=62, tag_tid=2, entry_tid=66742], [object_number=61, tag_tid=2, entry_tid=66741], [object_number=60, tag_tid=2, entry_tid=66740], [object_number=63, tag_tid=2, entry_tid=66739]]], state=connected] 2026-02-22T19:55:10.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=62, tag_tid=2, entry_tid=66742' 2026-02-22T19:55:10.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=62, tag_tid=2, entry_tid=66742' 2026-02-22T19:55:10.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=62, tag_tid=2, entry_tid=66742' '!=' 'object_number=60, tag_tid=2, entry_tid=66500' 2026-02-22T19:55:10.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:10.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=62, tag_tid=2, entry_tid=66742' = 'object_number=60, tag_tid=2, entry_tid=66500' 2026-02-22T19:55:10.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=62, tag_tid=2, entry_tid=66742' 2026-02-22T19:55:10.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:10.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:10.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:10.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=62, tag_tid=2, entry_tid=66742' 2026-02-22T19:55:10.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:10.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:10.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:10.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:10.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:10.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=62, tag_tid=2, entry_tid=66742' 2026-02-22T19:55:10.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:10.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=66742 2026-02-22T19:55:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 66742 2026-02-22T19:55:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:10.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:10.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:10.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:10.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:10.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:10.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:10.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:10.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:10.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:10.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:10.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=65, tag_tid=2, entry_tid=66905], [object_number=64, tag_tid=2, entry_tid=66904], [object_number=67, tag_tid=2, entry_tid=66903], [object_number=66, tag_tid=2, entry_tid=66902]]], state=connected] 2026-02-22T19:55:10.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:10.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:10.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:10.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:10.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:10.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:10.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:10.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:10.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:10.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:10.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=65, tag_tid=2, entry_tid=66905], [object_number=64, tag_tid=2, entry_tid=66904], [object_number=67, tag_tid=2, entry_tid=66903], [object_number=66, tag_tid=2, entry_tid=66902]]], state=connected] 2026-02-22T19:55:10.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=65, tag_tid=2, entry_tid=66905' 2026-02-22T19:55:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=65, tag_tid=2, entry_tid=66905' 2026-02-22T19:55:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=65, tag_tid=2, entry_tid=66905' '!=' 'object_number=62, tag_tid=2, entry_tid=66742' 2026-02-22T19:55:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=65, tag_tid=2, entry_tid=66905' = 'object_number=62, tag_tid=2, entry_tid=66742' 2026-02-22T19:55:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=65, tag_tid=2, entry_tid=66905' 2026-02-22T19:55:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:10.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=65, tag_tid=2, entry_tid=66905' 2026-02-22T19:55:10.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:10.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:10.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:10.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:10.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:10.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=65, tag_tid=2, entry_tid=66905' 2026-02-22T19:55:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=66905 2026-02-22T19:55:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 66905 2026-02-22T19:55:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:10.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:10.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:10.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:10.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:10.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:10.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:10.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:10.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:10.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:10.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:10.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:10.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:10.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:10.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:10.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:10.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=65, tag_tid=2, entry_tid=67157], [object_number=64, tag_tid=2, entry_tid=67156], [object_number=67, tag_tid=2, entry_tid=67155], [object_number=66, tag_tid=2, entry_tid=67154]]], state=connected] 2026-02-22T19:55:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:10.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:10.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:10.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:10.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:10.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:10.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:10.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:10.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:10.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:10.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:10.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:10.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:10.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:10.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=65, tag_tid=2, entry_tid=67157], [object_number=64, tag_tid=2, entry_tid=67156], [object_number=67, tag_tid=2, entry_tid=67155], [object_number=66, tag_tid=2, entry_tid=67154]]], state=connected] 2026-02-22T19:55:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=65, tag_tid=2, entry_tid=67157' 2026-02-22T19:55:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=65, tag_tid=2, entry_tid=67157' 2026-02-22T19:55:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=65, tag_tid=2, entry_tid=67157' '!=' 'object_number=65, tag_tid=2, entry_tid=66905' 2026-02-22T19:55:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=65, tag_tid=2, entry_tid=67157' = 'object_number=65, tag_tid=2, entry_tid=66905' 2026-02-22T19:55:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=65, tag_tid=2, entry_tid=67157' 2026-02-22T19:55:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=65, tag_tid=2, entry_tid=67157' 2026-02-22T19:55:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:10.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:10.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:10.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=65, tag_tid=2, entry_tid=67157' 2026-02-22T19:55:10.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:10.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:10.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=67157 2026-02-22T19:55:10.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 67157 2026-02-22T19:55:10.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:10.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:10.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:11.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:11.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:11.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:11.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:11.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:11.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:11.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:11.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:11.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:11.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:11.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:11.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:11.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:11.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:11.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:11.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:11.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:11.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:11.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:11.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:11.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:11.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:11.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:11.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:11.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:11.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:11.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:11.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:11.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:11.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:11.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:11.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:11.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:11.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=66, tag_tid=2, entry_tid=67398], [object_number=65, tag_tid=2, entry_tid=67397], [object_number=64, tag_tid=2, entry_tid=67396], [object_number=67, tag_tid=2, entry_tid=67395]]], state=connected] 2026-02-22T19:55:11.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:11.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:11.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:11.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:11.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:11.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:11.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:11.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:11.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:11.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=66, tag_tid=2, entry_tid=67398], [object_number=65, tag_tid=2, entry_tid=67397], [object_number=64, tag_tid=2, entry_tid=67396], [object_number=67, tag_tid=2, entry_tid=67395]]], state=connected] 2026-02-22T19:55:11.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=66, tag_tid=2, entry_tid=67398' 2026-02-22T19:55:11.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=66, tag_tid=2, entry_tid=67398' 2026-02-22T19:55:11.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=66, tag_tid=2, entry_tid=67398' '!=' 'object_number=65, tag_tid=2, entry_tid=67157' 2026-02-22T19:55:11.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:11.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=66, tag_tid=2, entry_tid=67398' = 'object_number=65, tag_tid=2, entry_tid=67157' 2026-02-22T19:55:11.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=66, tag_tid=2, entry_tid=67398' 2026-02-22T19:55:11.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:11.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:11.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:11.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:11.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=66, tag_tid=2, entry_tid=67398' 2026-02-22T19:55:11.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:11.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:11.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:11.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:11.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:11.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:11.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:11.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=66, tag_tid=2, entry_tid=67398' 2026-02-22T19:55:11.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:11.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=67398 2026-02-22T19:55:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 67398 2026-02-22T19:55:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:11.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:11.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:11.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:11.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:11.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:11.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:11.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:11.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:11.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:11.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:11.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:11.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:11.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:11.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:11.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:11.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:11.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:11.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:11.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:11.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:11.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:11.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=66, tag_tid=2, entry_tid=67622], [object_number=65, tag_tid=2, entry_tid=67621], [object_number=64, tag_tid=2, entry_tid=67620], [object_number=67, tag_tid=2, entry_tid=67619]]], state=connected] 2026-02-22T19:55:11.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:11.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:11.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:11.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:11.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:11.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:11.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:11.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=66, tag_tid=2, entry_tid=67622], [object_number=65, tag_tid=2, entry_tid=67621], [object_number=64, tag_tid=2, entry_tid=67620], [object_number=67, tag_tid=2, entry_tid=67619]]], state=connected] 2026-02-22T19:55:11.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=66, tag_tid=2, entry_tid=67622' 2026-02-22T19:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=66, tag_tid=2, entry_tid=67622' 2026-02-22T19:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=66, tag_tid=2, entry_tid=67622' '!=' 'object_number=66, tag_tid=2, entry_tid=67398' 2026-02-22T19:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=66, tag_tid=2, entry_tid=67622' = 'object_number=66, tag_tid=2, entry_tid=67398' 2026-02-22T19:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=66, tag_tid=2, entry_tid=67622' 2026-02-22T19:55:11.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:11.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:11.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=66, tag_tid=2, entry_tid=67622' 2026-02-22T19:55:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:11.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:11.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:11.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=66, tag_tid=2, entry_tid=67622' 2026-02-22T19:55:11.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:11.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=67622 2026-02-22T19:55:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 67622 2026-02-22T19:55:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:11.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:11.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:11.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:11.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:11.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:11.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:11.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:11.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:11.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:11.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:11.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:11.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:11.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:11.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:11.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:11.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:11.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:11.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:11.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:11.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:11.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:11.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:11.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:11.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:11.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:11.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:11.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=67888], [object_number=67, tag_tid=2, entry_tid=67887], [object_number=66, tag_tid=2, entry_tid=67886], [object_number=65, tag_tid=2, entry_tid=67885]]], state=connected] 2026-02-22T19:55:12.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:12.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:12.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:12.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:12.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:12.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:12.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=67888], [object_number=67, tag_tid=2, entry_tid=67887], [object_number=66, tag_tid=2, entry_tid=67886], [object_number=65, tag_tid=2, entry_tid=67885]]], state=connected] 2026-02-22T19:55:12.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=64, tag_tid=2, entry_tid=67888' 2026-02-22T19:55:12.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=64, tag_tid=2, entry_tid=67888' 2026-02-22T19:55:12.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=67888' '!=' 'object_number=66, tag_tid=2, entry_tid=67622' 2026-02-22T19:55:12.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:12.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=67888' = 'object_number=66, tag_tid=2, entry_tid=67622' 2026-02-22T19:55:12.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=64, tag_tid=2, entry_tid=67888' 2026-02-22T19:55:12.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:12.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:12.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:12.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=67888' 2026-02-22T19:55:12.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:12.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:12.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:12.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:12.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:12.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=67888' 2026-02-22T19:55:12.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:12.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=67888 2026-02-22T19:55:12.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 67888 2026-02-22T19:55:12.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:12.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:12.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:12.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:12.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:12.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:12.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:12.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:12.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:12.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:12.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:12.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:12.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:12.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:12.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:12.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:12.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:12.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:12.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:12.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:12.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:12.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:12.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:12.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:12.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:12.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:12.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:12.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:12.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:12.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:12.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:12.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:12.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:12.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:12.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:12.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:12.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:12.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:12.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:12.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=68140], [object_number=67, tag_tid=2, entry_tid=68139], [object_number=66, tag_tid=2, entry_tid=68138], [object_number=65, tag_tid=2, entry_tid=68137]]], state=connected] 2026-02-22T19:55:12.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:12.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:12.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:12.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:12.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:12.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:12.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:12.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:12.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:12.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:12.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:12.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:12.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=68140], [object_number=67, tag_tid=2, entry_tid=68139], [object_number=66, tag_tid=2, entry_tid=68138], [object_number=65, tag_tid=2, entry_tid=68137]]], state=connected] 2026-02-22T19:55:12.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=64, tag_tid=2, entry_tid=68140' 2026-02-22T19:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=64, tag_tid=2, entry_tid=68140' 2026-02-22T19:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=68140' '!=' 'object_number=64, tag_tid=2, entry_tid=67888' 2026-02-22T19:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=68140' = 'object_number=64, tag_tid=2, entry_tid=67888' 2026-02-22T19:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=64, tag_tid=2, entry_tid=68140' 2026-02-22T19:55:12.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:12.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:12.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:12.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=68140' 2026-02-22T19:55:12.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:12.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:12.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:12.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:12.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:12.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=68140' 2026-02-22T19:55:12.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:12.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=68140 2026-02-22T19:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 68140 2026-02-22T19:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:12.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:12.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:12.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:12.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:12.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:12.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:12.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:12.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:12.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:12.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:12.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:12.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:12.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:12.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:12.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:12.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:12.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:12.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:12.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:12.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:12.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:12.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:12.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:12.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:12.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:12.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:12.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:12.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=67, tag_tid=2, entry_tid=68375], [object_number=66, tag_tid=2, entry_tid=68374], [object_number=65, tag_tid=2, entry_tid=68373], [object_number=64, tag_tid=2, entry_tid=68372]]], state=connected] 2026-02-22T19:55:12.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:12.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:12.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:12.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:12.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:12.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:12.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:12.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:12.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=67, tag_tid=2, entry_tid=68375], [object_number=66, tag_tid=2, entry_tid=68374], [object_number=65, tag_tid=2, entry_tid=68373], [object_number=64, tag_tid=2, entry_tid=68372]]], state=connected] 2026-02-22T19:55:12.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:12.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=67, tag_tid=2, entry_tid=68375' 2026-02-22T19:55:12.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=67, tag_tid=2, entry_tid=68375' 2026-02-22T19:55:12.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=67, tag_tid=2, entry_tid=68375' '!=' 'object_number=64, tag_tid=2, entry_tid=68140' 2026-02-22T19:55:12.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:12.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=67, tag_tid=2, entry_tid=68375' = 'object_number=64, tag_tid=2, entry_tid=68140' 2026-02-22T19:55:12.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=67, tag_tid=2, entry_tid=68375' 2026-02-22T19:55:12.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:12.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:12.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:12.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=67, tag_tid=2, entry_tid=68375' 2026-02-22T19:55:12.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:12.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:12.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:12.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=67, tag_tid=2, entry_tid=68375' 2026-02-22T19:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=68375 2026-02-22T19:55:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 68375 2026-02-22T19:55:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:12.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:12.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:12.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:13.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:13.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:13.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:13.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:13.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:13.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:13.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:13.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:13.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:13.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=68600], [object_number=67, tag_tid=2, entry_tid=68599], [object_number=66, tag_tid=2, entry_tid=68598], [object_number=65, tag_tid=2, entry_tid=68597]]], state=connected] 2026-02-22T19:55:13.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:13.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:13.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:13.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:13.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:13.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:13.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=68600], [object_number=67, tag_tid=2, entry_tid=68599], [object_number=66, tag_tid=2, entry_tid=68598], [object_number=65, tag_tid=2, entry_tid=68597]]], state=connected] 2026-02-22T19:55:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=64, tag_tid=2, entry_tid=68600' 2026-02-22T19:55:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=64, tag_tid=2, entry_tid=68600' 2026-02-22T19:55:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=68600' '!=' 'object_number=67, tag_tid=2, entry_tid=68375' 2026-02-22T19:55:13.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:13.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=68600' = 'object_number=67, tag_tid=2, entry_tid=68375' 2026-02-22T19:55:13.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=64, tag_tid=2, entry_tid=68600' 2026-02-22T19:55:13.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:13.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:13.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=68600' 2026-02-22T19:55:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:13.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=68600' 2026-02-22T19:55:13.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:13.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=68600 2026-02-22T19:55:13.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 68600 2026-02-22T19:55:13.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:13.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:13.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:13.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:13.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:13.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:13.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:13.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:13.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:13.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:13.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:13.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:13.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:13.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:13.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:13.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:13.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:13.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:13.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:13.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:13.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:13.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:13.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:13.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:13.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:13.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:13.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:13.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:13.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:13.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:13.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:13.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:13.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:13.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=67, tag_tid=2, entry_tid=68823], [object_number=66, tag_tid=2, entry_tid=68822], [object_number=65, tag_tid=2, entry_tid=68821], [object_number=64, tag_tid=2, entry_tid=68820]]], state=connected] 2026-02-22T19:55:13.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:13.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:13.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:13.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:13.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:13.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:13.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:13.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=67, tag_tid=2, entry_tid=68823], [object_number=66, tag_tid=2, entry_tid=68822], [object_number=65, tag_tid=2, entry_tid=68821], [object_number=64, tag_tid=2, entry_tid=68820]]], state=connected] 2026-02-22T19:55:13.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=67, tag_tid=2, entry_tid=68823' 2026-02-22T19:55:13.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=67, tag_tid=2, entry_tid=68823' 2026-02-22T19:55:13.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=67, tag_tid=2, entry_tid=68823' '!=' 'object_number=64, tag_tid=2, entry_tid=68600' 2026-02-22T19:55:13.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:13.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=67, tag_tid=2, entry_tid=68823' = 'object_number=64, tag_tid=2, entry_tid=68600' 2026-02-22T19:55:13.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=67, tag_tid=2, entry_tid=68823' 2026-02-22T19:55:13.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:13.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:13.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:13.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=67, tag_tid=2, entry_tid=68823' 2026-02-22T19:55:13.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:13.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:13.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:13.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:13.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:13.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=67, tag_tid=2, entry_tid=68823' 2026-02-22T19:55:13.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:13.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=68823 2026-02-22T19:55:13.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 68823 2026-02-22T19:55:13.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:13.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:13.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:13.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:13.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:13.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:13.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:13.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:13.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:13.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:13.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:13.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:13.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:13.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:13.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:13.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:13.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:13.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:13.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:13.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:13.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:13.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:13.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:13.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:13.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:13.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:13.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:13.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:13.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:13.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:13.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:13.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:13.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:13.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:13.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:13.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:13.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:13.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:13.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=69048], [object_number=67, tag_tid=2, entry_tid=69047], [object_number=66, tag_tid=2, entry_tid=69046], [object_number=65, tag_tid=2, entry_tid=69045]]], state=connected] 2026-02-22T19:55:13.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:13.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:13.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:13.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:13.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:13.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:13.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:13.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:13.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=69048], [object_number=67, tag_tid=2, entry_tid=69047], [object_number=66, tag_tid=2, entry_tid=69046], [object_number=65, tag_tid=2, entry_tid=69045]]], state=connected] 2026-02-22T19:55:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=64, tag_tid=2, entry_tid=69048' 2026-02-22T19:55:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=64, tag_tid=2, entry_tid=69048' 2026-02-22T19:55:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=69048' '!=' 'object_number=67, tag_tid=2, entry_tid=68823' 2026-02-22T19:55:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:13.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=69048' = 'object_number=67, tag_tid=2, entry_tid=68823' 2026-02-22T19:55:13.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=64, tag_tid=2, entry_tid=69048' 2026-02-22T19:55:13.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:13.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:13.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:13.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=69048' 2026-02-22T19:55:13.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:13.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:13.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=69048' 2026-02-22T19:55:13.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:13.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:13.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=69048 2026-02-22T19:55:13.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 69048 2026-02-22T19:55:13.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:13.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:13.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:14.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:14.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:14.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:14.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:14.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:14.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:14.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:14.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:14.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:14.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=67, tag_tid=2, entry_tid=69303], [object_number=66, tag_tid=2, entry_tid=69302], [object_number=65, tag_tid=2, entry_tid=69301], [object_number=64, tag_tid=2, entry_tid=69300]]], state=connected] 2026-02-22T19:55:14.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=67, tag_tid=2, entry_tid=69303], [object_number=66, tag_tid=2, entry_tid=69302], [object_number=65, tag_tid=2, entry_tid=69301], [object_number=64, tag_tid=2, entry_tid=69300]]], state=connected] 2026-02-22T19:55:14.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=67, tag_tid=2, entry_tid=69303' 2026-02-22T19:55:14.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=67, tag_tid=2, entry_tid=69303' 2026-02-22T19:55:14.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=67, tag_tid=2, entry_tid=69303' '!=' 'object_number=64, tag_tid=2, entry_tid=69048' 2026-02-22T19:55:14.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:14.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=67, tag_tid=2, entry_tid=69303' = 'object_number=64, tag_tid=2, entry_tid=69048' 2026-02-22T19:55:14.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=67, tag_tid=2, entry_tid=69303' 2026-02-22T19:55:14.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:14.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:14.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:14.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=67, tag_tid=2, entry_tid=69303' 2026-02-22T19:55:14.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:14.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:14.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=67, tag_tid=2, entry_tid=69303' 2026-02-22T19:55:14.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:14.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=69303 2026-02-22T19:55:14.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 69303 2026-02-22T19:55:14.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:14.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:14.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:14.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:14.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:14.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:14.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:14.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:14.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:14.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:14.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:14.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:14.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:14.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:14.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:14.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:14.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:14.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:14.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:14.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:14.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:14.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:14.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:14.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:14.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:14.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:14.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:14.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:14.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:14.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:14.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:14.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:14.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:14.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:14.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:14.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:14.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:14.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:14.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:14.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=65, tag_tid=2, entry_tid=69561], [object_number=64, tag_tid=2, entry_tid=69560], [object_number=67, tag_tid=2, entry_tid=69559], [object_number=66, tag_tid=2, entry_tid=69558]]], state=connected] 2026-02-22T19:55:14.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:14.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:14.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:14.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:14.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:14.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=65, tag_tid=2, entry_tid=69561], [object_number=64, tag_tid=2, entry_tid=69560], [object_number=67, tag_tid=2, entry_tid=69559], [object_number=66, tag_tid=2, entry_tid=69558]]], state=connected] 2026-02-22T19:55:14.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=65, tag_tid=2, entry_tid=69561' 2026-02-22T19:55:14.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=65, tag_tid=2, entry_tid=69561' 2026-02-22T19:55:14.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=65, tag_tid=2, entry_tid=69561' '!=' 'object_number=67, tag_tid=2, entry_tid=69303' 2026-02-22T19:55:14.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:14.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=65, tag_tid=2, entry_tid=69561' = 'object_number=67, tag_tid=2, entry_tid=69303' 2026-02-22T19:55:14.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=65, tag_tid=2, entry_tid=69561' 2026-02-22T19:55:14.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:14.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:14.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:14.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=65, tag_tid=2, entry_tid=69561' 2026-02-22T19:55:14.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:14.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:14.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=65, tag_tid=2, entry_tid=69561' 2026-02-22T19:55:14.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:14.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=69561 2026-02-22T19:55:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 69561 2026-02-22T19:55:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:14.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:14.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:14.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:14.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:14.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:14.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:14.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:14.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:14.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:14.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:14.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:14.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:14.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:14.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:14.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:14.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:14.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:14.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:14.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:14.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:14.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:14.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:14.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:14.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:14.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:14.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:14.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=65, tag_tid=2, entry_tid=69805], [object_number=64, tag_tid=2, entry_tid=69804], [object_number=67, tag_tid=2, entry_tid=69803], [object_number=66, tag_tid=2, entry_tid=69802]]], state=connected] 2026-02-22T19:55:14.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:14.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:14.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:14.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:14.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:14.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:14.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:14.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:14.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=65, tag_tid=2, entry_tid=69805], [object_number=64, tag_tid=2, entry_tid=69804], [object_number=67, tag_tid=2, entry_tid=69803], [object_number=66, tag_tid=2, entry_tid=69802]]], state=connected] 2026-02-22T19:55:14.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:14.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=65, tag_tid=2, entry_tid=69805' 2026-02-22T19:55:14.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=65, tag_tid=2, entry_tid=69805' 2026-02-22T19:55:14.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=65, tag_tid=2, entry_tid=69805' '!=' 'object_number=65, tag_tid=2, entry_tid=69561' 2026-02-22T19:55:14.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:14.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=65, tag_tid=2, entry_tid=69805' = 'object_number=65, tag_tid=2, entry_tid=69561' 2026-02-22T19:55:14.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=65, tag_tid=2, entry_tid=69805' 2026-02-22T19:55:14.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:14.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:14.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:14.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=65, tag_tid=2, entry_tid=69805' 2026-02-22T19:55:14.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:14.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:14.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:14.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:14.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:14.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=65, tag_tid=2, entry_tid=69805' 2026-02-22T19:55:14.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:14.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=69805 2026-02-22T19:55:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 69805 2026-02-22T19:55:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:15.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:15.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:15.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:15.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:15.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:15.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:15.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:15.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:15.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:15.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:15.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=66, tag_tid=2, entry_tid=70034], [object_number=65, tag_tid=2, entry_tid=70033], [object_number=64, tag_tid=2, entry_tid=70032], [object_number=67, tag_tid=2, entry_tid=70031]]], state=connected] 2026-02-22T19:55:15.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:15.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:15.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:15.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:15.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:15.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:15.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:15.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=66, tag_tid=2, entry_tid=70034], [object_number=65, tag_tid=2, entry_tid=70033], [object_number=64, tag_tid=2, entry_tid=70032], [object_number=67, tag_tid=2, entry_tid=70031]]], state=connected] 2026-02-22T19:55:15.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=66, tag_tid=2, entry_tid=70034' 2026-02-22T19:55:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=66, tag_tid=2, entry_tid=70034' 2026-02-22T19:55:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=66, tag_tid=2, entry_tid=70034' '!=' 'object_number=65, tag_tid=2, entry_tid=69805' 2026-02-22T19:55:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=66, tag_tid=2, entry_tid=70034' = 'object_number=65, tag_tid=2, entry_tid=69805' 2026-02-22T19:55:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=66, tag_tid=2, entry_tid=70034' 2026-02-22T19:55:15.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:15.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:15.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:15.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:15.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=66, tag_tid=2, entry_tid=70034' 2026-02-22T19:55:15.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:15.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:15.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=66, tag_tid=2, entry_tid=70034' 2026-02-22T19:55:15.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:15.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:15.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=70034 2026-02-22T19:55:15.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 70034 2026-02-22T19:55:15.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:15.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:15.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:15.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:15.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:15.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:15.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:15.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:15.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:15.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:15.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:15.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:15.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:15.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:15.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:15.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:15.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:15.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:15.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:15.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:15.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:15.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:15.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:15.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:15.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=70324], [object_number=67, tag_tid=2, entry_tid=70323], [object_number=66, tag_tid=2, entry_tid=70322], [object_number=65, tag_tid=2, entry_tid=70321]]], state=connected] 2026-02-22T19:55:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:15.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:15.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:15.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:15.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:15.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:15.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:15.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=70324], [object_number=67, tag_tid=2, entry_tid=70323], [object_number=66, tag_tid=2, entry_tid=70322], [object_number=65, tag_tid=2, entry_tid=70321]]], state=connected] 2026-02-22T19:55:15.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=64, tag_tid=2, entry_tid=70324' 2026-02-22T19:55:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=64, tag_tid=2, entry_tid=70324' 2026-02-22T19:55:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=70324' '!=' 'object_number=66, tag_tid=2, entry_tid=70034' 2026-02-22T19:55:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=70324' = 'object_number=66, tag_tid=2, entry_tid=70034' 2026-02-22T19:55:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=64, tag_tid=2, entry_tid=70324' 2026-02-22T19:55:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:15.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=70324' 2026-02-22T19:55:15.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:15.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:15.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:15.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=70324' 2026-02-22T19:55:15.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:15.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=70324 2026-02-22T19:55:15.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 70324 2026-02-22T19:55:15.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:15.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:15.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:15.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:15.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:15.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:15.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:15.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:15.961 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5707452271373085) > weight (0.25) 2026-02-22T19:55:15.961 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-22T19:55:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-22T19:55:15.962 INFO:tasks.rbd_mirror_thrash:waiting for 2.0976037181626306 secs before reviving daemons 2026-02-22T19:55:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T19:55:15.954+0000 7f544fc63640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 19980) UID: 1000 2026-02-22T19:55:15.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:15.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:15.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:15.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:15.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:15.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:15.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=70584], [object_number=67, tag_tid=2, entry_tid=70583], [object_number=66, tag_tid=2, entry_tid=70582], [object_number=65, tag_tid=2, entry_tid=70581]]], state=connected] 2026-02-22T19:55:16.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:16.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:16.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:16.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:16.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:16.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:16.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:16.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:16.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:16.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:16.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:16.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:16.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=70584], [object_number=67, tag_tid=2, entry_tid=70583], [object_number=66, tag_tid=2, entry_tid=70582], [object_number=65, tag_tid=2, entry_tid=70581]]], state=connected] 2026-02-22T19:55:16.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=64, tag_tid=2, entry_tid=70584' 2026-02-22T19:55:16.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=64, tag_tid=2, entry_tid=70584' 2026-02-22T19:55:16.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=70584' '!=' 'object_number=64, tag_tid=2, entry_tid=70324' 2026-02-22T19:55:16.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:16.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=70584' = 'object_number=64, tag_tid=2, entry_tid=70324' 2026-02-22T19:55:16.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=64, tag_tid=2, entry_tid=70584' 2026-02-22T19:55:16.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:16.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:16.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:16.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=70584' 2026-02-22T19:55:16.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:16.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:16.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:16.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:16.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:16.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=70584' 2026-02-22T19:55:16.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:16.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=70584 2026-02-22T19:55:16.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 70584 2026-02-22T19:55:16.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:16.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:16.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:16.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:16.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:16.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:16.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:16.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:16.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:16.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:16.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:16.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:16.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:16.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:16.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:16.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:16.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:16.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:16.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:16.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:16.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:16.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:16.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:16.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:16.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:16.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:16.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:16.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:16.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:16.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:16.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=70844], [object_number=67, tag_tid=2, entry_tid=70843], [object_number=66, tag_tid=2, entry_tid=70842], [object_number=65, tag_tid=2, entry_tid=70841]]], state=connected] 2026-02-22T19:55:16.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:16.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:16.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:16.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 16 2026-02-22T19:55:16.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:16.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:16.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:16.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=64, tag_tid=2, entry_tid=70844], [object_number=67, tag_tid=2, entry_tid=70843], [object_number=66, tag_tid=2, entry_tid=70842], [object_number=65, tag_tid=2, entry_tid=70841]]], state=connected] 2026-02-22T19:55:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=64, tag_tid=2, entry_tid=70844' 2026-02-22T19:55:16.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=64, tag_tid=2, entry_tid=70844' 2026-02-22T19:55:16.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=70844' '!=' 'object_number=64, tag_tid=2, entry_tid=70584' 2026-02-22T19:55:16.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:16.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=64, tag_tid=2, entry_tid=70844' = 'object_number=64, tag_tid=2, entry_tid=70584' 2026-02-22T19:55:16.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=64, tag_tid=2, entry_tid=70844' 2026-02-22T19:55:16.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:16.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:16.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:16.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=70844' 2026-02-22T19:55:16.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:16.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:16.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:16.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:16.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:16.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=64, tag_tid=2, entry_tid=70844' 2026-02-22T19:55:16.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:16.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=70844 2026-02-22T19:55:16.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 70844 2026-02-22T19:55:16.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:16.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:16.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:16.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:16.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:16.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:16.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:16.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:16.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:16.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:16.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:16.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:16.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:16.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:16.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:16.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=71065], [object_number=68, tag_tid=2, entry_tid=71064], [object_number=71, tag_tid=2, entry_tid=71063], [object_number=70, tag_tid=2, entry_tid=71062]]], state=connected] 2026-02-22T19:55:16.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:16.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:16.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:16.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:16.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:16.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:16.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=71065], [object_number=68, tag_tid=2, entry_tid=71064], [object_number=71, tag_tid=2, entry_tid=71063], [object_number=70, tag_tid=2, entry_tid=71062]]], state=connected] 2026-02-22T19:55:16.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=69, tag_tid=2, entry_tid=71065' 2026-02-22T19:55:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=69, tag_tid=2, entry_tid=71065' 2026-02-22T19:55:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=71065' '!=' 'object_number=64, tag_tid=2, entry_tid=70844' 2026-02-22T19:55:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=71065' = 'object_number=64, tag_tid=2, entry_tid=70844' 2026-02-22T19:55:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=69, tag_tid=2, entry_tid=71065' 2026-02-22T19:55:16.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:16.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:16.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:16.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=2, entry_tid=71065' 2026-02-22T19:55:16.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:16.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:16.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:16.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:16.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:16.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=2, entry_tid=71065' 2026-02-22T19:55:16.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:16.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:16.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=71065 2026-02-22T19:55:16.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 71065 2026-02-22T19:55:16.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:16.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:16.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:16.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:16.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:16.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:16.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:16.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:16.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:16.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:16.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:16.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:16.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:16.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:16.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:17.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:17.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:17.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:17.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:17.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:17.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:17.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:17.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=68, tag_tid=2, entry_tid=71312], [object_number=71, tag_tid=2, entry_tid=71311], [object_number=70, tag_tid=2, entry_tid=71310], [object_number=69, tag_tid=2, entry_tid=71309]]], state=connected] 2026-02-22T19:55:17.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:17.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:17.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:17.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:17.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:17.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:17.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:17.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:17.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=68, tag_tid=2, entry_tid=71312], [object_number=71, tag_tid=2, entry_tid=71311], [object_number=70, tag_tid=2, entry_tid=71310], [object_number=69, tag_tid=2, entry_tid=71309]]], state=connected] 2026-02-22T19:55:17.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=68, tag_tid=2, entry_tid=71312' 2026-02-22T19:55:17.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=68, tag_tid=2, entry_tid=71312' 2026-02-22T19:55:17.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=68, tag_tid=2, entry_tid=71312' '!=' 'object_number=69, tag_tid=2, entry_tid=71065' 2026-02-22T19:55:17.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:17.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=68, tag_tid=2, entry_tid=71312' = 'object_number=69, tag_tid=2, entry_tid=71065' 2026-02-22T19:55:17.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=68, tag_tid=2, entry_tid=71312' 2026-02-22T19:55:17.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:17.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:17.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:17.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:17.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=68, tag_tid=2, entry_tid=71312' 2026-02-22T19:55:17.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:17.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:17.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:17.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:17.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:17.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=68, tag_tid=2, entry_tid=71312' 2026-02-22T19:55:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:17.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=71312 2026-02-22T19:55:17.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 71312 2026-02-22T19:55:17.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:17.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:17.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:17.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:17.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:17.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:17.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:17.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:17.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:17.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:17.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:17.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:17.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:17.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:17.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:17.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:17.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:17.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:17.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=68, tag_tid=2, entry_tid=71596], [object_number=71, tag_tid=2, entry_tid=71595], [object_number=70, tag_tid=2, entry_tid=71594], [object_number=69, tag_tid=2, entry_tid=71593]]], state=connected] 2026-02-22T19:55:17.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:17.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:17.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:17.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:17.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:17.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:17.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=68, tag_tid=2, entry_tid=71596], [object_number=71, tag_tid=2, entry_tid=71595], [object_number=70, tag_tid=2, entry_tid=71594], [object_number=69, tag_tid=2, entry_tid=71593]]], state=connected] 2026-02-22T19:55:17.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=68, tag_tid=2, entry_tid=71596' 2026-02-22T19:55:17.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=68, tag_tid=2, entry_tid=71596' 2026-02-22T19:55:17.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=68, tag_tid=2, entry_tid=71596' '!=' 'object_number=68, tag_tid=2, entry_tid=71312' 2026-02-22T19:55:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=68, tag_tid=2, entry_tid=71596' = 'object_number=68, tag_tid=2, entry_tid=71312' 2026-02-22T19:55:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=68, tag_tid=2, entry_tid=71596' 2026-02-22T19:55:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:17.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:17.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:17.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=68, tag_tid=2, entry_tid=71596' 2026-02-22T19:55:17.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:17.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:17.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:17.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:17.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:17.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:17.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:17.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=68, tag_tid=2, entry_tid=71596' 2026-02-22T19:55:17.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:17.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:17.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=71596 2026-02-22T19:55:17.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 71596 2026-02-22T19:55:17.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:17.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:17.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:17.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:17.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:17.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:17.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:17.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:17.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:17.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:17.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:17.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:17.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:17.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:17.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:17.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:17.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:17.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:17.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:17.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:17.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:18.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:18.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:18.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:18.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=68, tag_tid=2, entry_tid=71988], [object_number=71, tag_tid=2, entry_tid=71987], [object_number=70, tag_tid=2, entry_tid=71986], [object_number=69, tag_tid=2, entry_tid=71985]]], state=connected] 2026-02-22T19:55:18.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:18.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:18.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:18.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:18.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=68, tag_tid=2, entry_tid=71988], [object_number=71, tag_tid=2, entry_tid=71987], [object_number=70, tag_tid=2, entry_tid=71986], [object_number=69, tag_tid=2, entry_tid=71985]]], state=connected] 2026-02-22T19:55:18.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=68, tag_tid=2, entry_tid=71988' 2026-02-22T19:55:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=68, tag_tid=2, entry_tid=71988' 2026-02-22T19:55:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=68, tag_tid=2, entry_tid=71988' '!=' 'object_number=68, tag_tid=2, entry_tid=71596' 2026-02-22T19:55:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=68, tag_tid=2, entry_tid=71988' = 'object_number=68, tag_tid=2, entry_tid=71596' 2026-02-22T19:55:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=68, tag_tid=2, entry_tid=71988' 2026-02-22T19:55:18.061 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-22T19:55:18.061 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T19:55:18.061 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-22T19:55:18.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-22T19:55:18.062 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-22T19:55:18.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:18.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:18.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=68, tag_tid=2, entry_tid=71988' 2026-02-22T19:55:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-22T19:55:18.065 INFO:tasks.rbd_mirror_thrash:waiting for 104.2077484023263 secs before thrashing 2026-02-22T19:55:18.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:18.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:18.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:18.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:18.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=68, tag_tid=2, entry_tid=71988' 2026-02-22T19:55:18.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:18.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=71988 2026-02-22T19:55:18.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 71988 2026-02-22T19:55:18.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:18.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:18.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:18.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:18.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:18.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:18.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:18.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:18.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:18.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:18.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:18.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:18.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:18.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:18.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:18.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:18.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:18.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:18.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:18.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:18.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:18.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=68, tag_tid=2, entry_tid=72212], [object_number=71, tag_tid=2, entry_tid=72211], [object_number=70, tag_tid=2, entry_tid=72210], [object_number=69, tag_tid=2, entry_tid=72209]]], state=connected] 2026-02-22T19:55:18.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:18.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:18.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:18.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:18.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:18.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:18.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:18.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:18.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:18.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=68, tag_tid=2, entry_tid=72212], [object_number=71, tag_tid=2, entry_tid=72211], [object_number=70, tag_tid=2, entry_tid=72210], [object_number=69, tag_tid=2, entry_tid=72209]]], state=connected] 2026-02-22T19:55:18.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=68, tag_tid=2, entry_tid=72212' 2026-02-22T19:55:18.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=68, tag_tid=2, entry_tid=72212' 2026-02-22T19:55:18.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=68, tag_tid=2, entry_tid=72212' '!=' 'object_number=68, tag_tid=2, entry_tid=71988' 2026-02-22T19:55:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=68, tag_tid=2, entry_tid=72212' = 'object_number=68, tag_tid=2, entry_tid=71988' 2026-02-22T19:55:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=68, tag_tid=2, entry_tid=72212' 2026-02-22T19:55:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:18.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:18.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:18.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=68, tag_tid=2, entry_tid=72212' 2026-02-22T19:55:18.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:18.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:18.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:18.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=68, tag_tid=2, entry_tid=72212' 2026-02-22T19:55:18.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:18.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=72212 2026-02-22T19:55:18.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 72212 2026-02-22T19:55:18.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:18.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:18.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:18.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:18.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:18.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:18.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:18.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:18.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:18.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:18.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:18.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:18.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:18.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:18.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:18.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:18.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:18.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:18.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72437], [object_number=68, tag_tid=2, entry_tid=72436], [object_number=71, tag_tid=2, entry_tid=72435], [object_number=70, tag_tid=2, entry_tid=72434]]], state=connected] 2026-02-22T19:55:18.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:18.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:18.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:18.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:18.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:18.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:18.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:18.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72437], [object_number=68, tag_tid=2, entry_tid=72436], [object_number=71, tag_tid=2, entry_tid=72435], [object_number=70, tag_tid=2, entry_tid=72434]]], state=connected] 2026-02-22T19:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:18.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=69, tag_tid=2, entry_tid=72437' 2026-02-22T19:55:18.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=69, tag_tid=2, entry_tid=72437' 2026-02-22T19:55:18.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=72437' '!=' 'object_number=68, tag_tid=2, entry_tid=72212' 2026-02-22T19:55:18.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:18.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=72437' = 'object_number=68, tag_tid=2, entry_tid=72212' 2026-02-22T19:55:18.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=69, tag_tid=2, entry_tid=72437' 2026-02-22T19:55:18.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:18.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:18.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:18.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=2, entry_tid=72437' 2026-02-22T19:55:18.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:18.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:18.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:18.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:18.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:18.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=2, entry_tid=72437' 2026-02-22T19:55:18.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:18.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:18.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=72437 2026-02-22T19:55:18.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 72437 2026-02-22T19:55:18.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:18.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:18.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:18.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:18.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:18.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:18.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:18.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:18.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:18.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:18.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:18.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:19.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:19.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:19.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=72641' '!=' 'object_number=69, tag_tid=2, entry_tid=72437' 2026-02-22T19:55:19.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:55:19.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=72641' = 'object_number=69, tag_tid=2, entry_tid=72437' 2026-02-22T19:55:19.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:19.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:19.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:19.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:19.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T19:55:19.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:19.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:55:19.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:19.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T19:55:19.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:19.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:19.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:19.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:55:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:55:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:55:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=72641 2026-02-22T19:55:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 72641 2026-02-22T19:55:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:55:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:55:19.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:19.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:19.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:19.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:19.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:19.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:19.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:19.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:19.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:19.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:19.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:19.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:19.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:19.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:19.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:19.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:19.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:19.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:19.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:19.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:19.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:19.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:19.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:19.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:19.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:19.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:19.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:19.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:19.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:19.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:19.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:19.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:19.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:19.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:19.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:19.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:19.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:19.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=72641' '!=' 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T19:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:19.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:19.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:19.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:19.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:19.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:19.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:19.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:19.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:19.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:19.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:19.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:19.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:19.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:19.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:19.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:19.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=72641' '!=' 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T19:55:20.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:20.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:20.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:20.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:20.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:20.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:20.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:20.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:20.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:20.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:20.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:20.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:20.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:20.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:20.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:20.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:20.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:20.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:20.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=72641' '!=' 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T19:55:22.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:22.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:22.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:22.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:22.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:22.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:22.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:22.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:22.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:22.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:22.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:22.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:22.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:22.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:22.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:22.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:22.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:22.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:22.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:22.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:22.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:22.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=2, entry_tid=72641], [object_number=68, tag_tid=2, entry_tid=72640], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:22.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:22.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:22.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:22.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=69, tag_tid=2, entry_tid=72641' '!=' 'object_number=69, tag_tid=2, entry_tid=72641' 2026-02-22T19:55:22.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:22.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T19:55:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:55:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:24.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:55:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:55:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:55:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:24.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:55:24.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:55:24.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:55:24.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:55:24.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:24.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:55:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:55:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:55:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:24.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:55:24.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:24.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:24.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:24.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:24.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:24.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:24.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:24.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:24.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:24.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:55:24.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:24.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:55:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:55:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:55:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:55:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:55:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:24.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:55:24.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:24.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:24.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:55:24.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 17 2026-02-22T19:55:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 17 2026-02-22T19:55:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:55:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=69, tag_tid=3, entry_tid=1], [object_number=68, tag_tid=3, entry_tid=0], [object_number=71, tag_tid=2, entry_tid=72639], [object_number=70, tag_tid=2, entry_tid=72638]]], state=connected] 2026-02-22T19:55:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:55:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=69, tag_tid=3, entry_tid=1' -a 'object_number=69, tag_tid=3, entry_tid=1' = 'object_number=69, tag_tid=3, entry_tid=1' 2026-02-22T19:55:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2026-02-22T19:55:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap1 2026-02-22T19:55:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:55:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T19:55:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:55:25.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2026-02-22T19:55:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T19:55:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T19:55:25.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 19:54:36 2026 2026-02-22T19:55:25.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T19:55:25.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T19:55:25.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T19:55:25.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T19:55:25.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:55:25.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap1 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap1 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:55:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:55:25.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:55:26.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:55:26.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:55:26.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:55:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:55:27.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:55:27.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:27.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap1-clean 2026-02-22T19:55:27.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap1-clean 2026-02-22T19:55:27.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:55:27.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:27.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:27.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap1-clean 2026-02-22T19:55:27.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T19:55:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T19:55:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T19:55:27.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:55:27.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:27.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:27.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T19:55:27.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T19:55:27.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T19:55:27.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T19:55:27.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 31009792 bytes 2026-02-22T19:55:28.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 79064064 bytes 2026-02-22T19:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 127682560 bytes 2026-02-22T19:55:29.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 139435008 bytes 2026-02-22T19:55:30.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 154770432 bytes 2026-02-22T19:55:31.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 194958336 bytes 2026-02-22T19:55:31.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 272877568 bytes 2026-02-22T19:55:32.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 301059072 bytes 2026-02-22T19:55:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T19:55:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T19:55:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T19:55:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:55:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:55:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:55:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:33.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:55:33.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:55:33.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:55:33.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:55:33.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:55:33.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T19:55:33.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:55:33.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T19:55:33.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":8425610.630000001,"entries_behind_primary":0,"entries_per_second":523.67,"non_primary_position":{"entry_tid":1,"object_number":69,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":69,"tag_tid":3}} 2026-02-22T19:55:33.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.0 on trial066 2026-02-22T19:55:33.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:55:33 2026-02-22T19:55:33.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T19:55:33.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T19:55:33.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T19:55:33.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T19:55:33.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:55:33' 2026-02-22T19:55:33.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:55:33.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T19:55:33.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T19:55:33.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":8425610.630000001,"entries_behind_primary":0,"entries_per_second":523.67,"non_primary_position":{"entry_tid":1,"object_number":69,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":69,"tag_tid":3}} 2026-02-22T19:55:33.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:55:33.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:55:33.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T19:55:33.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:55:33.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:33.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.0 on trial066 2026-02-22T19:55:33.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T19:55:33.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:55:33' 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:55:33' 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 19:55:33' 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:33.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142024257mirror.0trial06619.2.3-689-gf90f4eaetrueOK42104263mirror.2trial06619.2.3-689-gf90f4eaefalseOK42134266mirror.3trial06619.2.3-689-gf90f4eaefalseOK43154318mirror.1trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":8425610.630000001,"entries_behind_primary":0,"entries_per_second":523.67,"non_primary_position":{"entry_tid":1,"object_number":69,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":69,"tag_tid":3}}42024257mirror.0trial0662026-02-22 19:55:33cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 19:55:33' 2026-02-22T19:55:33.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T19:55:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:33.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:55:33' 2026-02-22T19:55:33.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:33.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T19:55:33.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:55:33.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T19:55:33.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:55:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:55:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap2 2026-02-22T19:55:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap2 2026-02-22T19:55:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:55:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:55:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:55:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap2 2026-02-22T19:55:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2026-02-22T19:56:06.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:56:07.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:56:07.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:07.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:07.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:07.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T19:56:07.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T19:56:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T19:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T19:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T19:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T19:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T19:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T19:56:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:56:07.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:07.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:56:07.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:56:07.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:56:07.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:56:07.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:07.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:56:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:07.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:07.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:07.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:07.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:07.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T19:56:07.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:56:07.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:56:07.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T19:56:07.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T19:56:07.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:56:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:56:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:56:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:56:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:56:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:56:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:56:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:56:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:56:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:56:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:56:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:07.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:07.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:07.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:07.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:56:07.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:56:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:07.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:07.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:07.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:07.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:07.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:07.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:07.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:07.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:07.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:07.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:56:07.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:56:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:56:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:07.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:07.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:07.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:07.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:07.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:08.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:08.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:08.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:08.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:08.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:08.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=89, tag_tid=6, entry_tid=1' -a 'object_number=89, tag_tid=6, entry_tid=1' = 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=86, tag_tid=5, entry_tid=19214' '!=' '' 2026-02-22T19:56:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:56:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=86, tag_tid=5, entry_tid=19214' = '' 2026-02-22T19:56:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:08.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:08.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:56:08.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:56:08.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=6 2026-02-22T19:56:08.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:08.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:56:08.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:56:08.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=5 2026-02-22T19:56:08.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:08.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:56:08.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:56:08.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:56:08.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:08.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:56:08.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:56:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=19214 2026-02-22T19:56:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 6 = 5 -a 1 -le 19214 2026-02-22T19:56:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:56:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:56:08.233 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.40201720962332854) > weight (0.25) 2026-02-22T19:56:08.233 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.40183838780338843) > weight (0.25) 2026-02-22T19:56:08.233 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7843375312782606) > weight (0.25) 2026-02-22T19:56:08.233 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9519741477331377) > weight (0.25) 2026-02-22T19:56:08.233 INFO:tasks.rbd_mirror_thrash:waiting for 119.12532851343391 secs before thrashing 2026-02-22T19:56:08.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:56:08.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:56:08.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:56:08.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:08.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:56:08.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:56:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:56:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:56:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:56:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:56:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:56:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:56:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:56:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:56:08.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:08.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:56:08.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:08.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:08.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:08.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:56:08.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:08.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:08.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:08.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:56:08.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:08.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:08.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:08.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:08.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:08.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:08.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:08.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:08.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:08.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:08.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:08.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:08.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:08.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:56:08.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:08.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:08.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:08.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:56:08.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:08.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:08.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:08.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:56:08.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:08.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:08.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:08.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:08.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:08.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:08.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:08.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:08.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:08.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:08.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:08.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:08.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=89, tag_tid=6, entry_tid=1' -a 'object_number=89, tag_tid=6, entry_tid=1' = 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=86, tag_tid=5, entry_tid=19214' '!=' 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T19:56:08.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:56:08.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:08.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:56:08.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:56:08.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:08.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:56:08.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:08.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:56:08.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:08.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:56:08.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:56:08.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:08.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:56:08.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:08.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:56:08.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:56:08.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:56:08.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:08.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:08.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:08.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:08.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:56:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:56:09.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:09.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:09.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:09.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:56:09.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:09.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:09.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:09.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:56:09.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:09.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:09.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:09.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:09.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:09.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:09.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:09.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:09.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:09.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:09.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:56:09.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:09.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:09.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:09.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:56:09.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:56:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:09.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:09.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:09.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:09.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:09.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:09.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:09.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:09.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=89, tag_tid=6, entry_tid=1' -a 'object_number=89, tag_tid=6, entry_tid=1' = 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:09.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=86, tag_tid=5, entry_tid=19214' '!=' 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:09.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:09.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:56:09.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:09.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:56:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:56:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:56:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:56:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:56:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:56:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:56:10.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:10.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:56:10.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:10.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:10.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:10.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:56:10.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:10.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:10.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:10.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:56:10.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:10.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:10.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:10.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:10.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:10.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:10.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:10.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:10.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:10.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:10.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:10.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:10.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:10.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:56:10.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:10.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:10.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:10.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:56:10.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:10.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:10.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:10.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:56:10.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:10.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:10.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:10.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:10.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=89, tag_tid=6, entry_tid=1' -a 'object_number=89, tag_tid=6, entry_tid=1' = 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:10.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=86, tag_tid=5, entry_tid=19214' '!=' 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:10.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:10.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T19:56:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:56:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:56:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:56:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:56:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:56:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:56:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:56:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:56:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:56:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:11.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:56:11.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:56:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:56:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:56:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:56:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:56:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:11.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:11.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:11.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:11.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:56:11.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:56:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:56:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:56:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:11.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:11.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:11.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:11.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:11.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:11.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:56:11.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:11.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:11.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:11.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:56:11.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:11.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:11.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:11.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:56:11.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:11.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:11.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:11.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:11.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:11.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:11.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:11.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:11.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:11.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:11.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=89, tag_tid=6, entry_tid=1' -a 'object_number=89, tag_tid=6, entry_tid=1' = 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=86, tag_tid=5, entry_tid=19214' '!=' 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T19:56:13.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:56:13.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:13.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:13.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:56:13.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:56:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:56:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:56:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:56:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:56:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:56:13.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:56:13.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:13.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:56:13.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:13.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:13.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:13.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:56:13.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:56:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:13.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:13.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:13.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:13.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:56:13.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:13.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:13.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=86, tag_tid=5, entry_tid=19214], [object_number=85, tag_tid=5, entry_tid=19213], [object_number=84, tag_tid=5, entry_tid=19212], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:13.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=89, tag_tid=6, entry_tid=1' -a 'object_number=89, tag_tid=6, entry_tid=1' = 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=86, tag_tid=5, entry_tid=19214' '!=' 'object_number=86, tag_tid=5, entry_tid=19214' 2026-02-22T19:56:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T19:56:15.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:56:15.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:15.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:15.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:15.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:56:15.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:56:15.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:56:16.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:56:16.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:56:16.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:16.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:16.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:56:16.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:56:16.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:16.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:56:16.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:16.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:56:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:56:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:56:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:56:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:56:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:56:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:56:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:56:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:16.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:16.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:16.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:16.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:56:16.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:16.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:56:16.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:16.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:16.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:16.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:56:16.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:16.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:16.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:16.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:56:16.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:16.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:16.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:16.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:16.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:16.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:16.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:16.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:16.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:16.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:16.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:16.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:56:16.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:16.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:16.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:16.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:56:16.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:56:16.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:56:16.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:56:16.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:56:16.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:16.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:56:16.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:16.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:16.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:56:16.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 21 2026-02-22T19:56:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 22 2026-02-22T19:56:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:56:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=89, tag_tid=6, entry_tid=1], [object_number=88, tag_tid=6, entry_tid=0], [object_number=86, tag_tid=5, entry_tid=19214], [object_number=87, tag_tid=5, entry_tid=19211]]], state=connected] 2026-02-22T19:56:16.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:56:16.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=89, tag_tid=6, entry_tid=1' -a 'object_number=89, tag_tid=6, entry_tid=1' = 'object_number=89, tag_tid=6, entry_tid=1' 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap2 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T19:56:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:56:17.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2026-02-22T19:56:17.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 3 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T19:56:17.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T19:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 19:55:26 2026 2026-02-22T19:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 19:55:37 2026 2026-02-22T19:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T19:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T19:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T19:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T19:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T19:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n snap1-clean ']' 2026-02-22T19:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap1-clean 2026-02-22T19:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap1-clean 2026-02-22T19:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T19:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:56:17.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:56:17.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:18.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:56:18.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:18.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap2 2026-02-22T19:56:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap2 2026-02-22T19:56:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T19:56:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:56:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:56:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:56:18.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:19.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:19.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:56:19.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:19.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap2-clean 2026-02-22T19:56:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap2-clean 2026-02-22T19:56:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:56:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap2-clean 2026-02-22T19:56:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T19:56:20.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T19:56:20.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T19:56:20.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:56:20.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:20.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:20.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T19:56:20.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=7 2026-02-22T19:56:20.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T19:56:20.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T19:56:21.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 51555328 bytes 2026-02-22T19:56:21.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 70658048 bytes 2026-02-22T19:56:22.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 99478528 bytes 2026-02-22T19:56:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 123944960 bytes 2026-02-22T19:56:23.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 174112768 bytes 2026-02-22T19:56:23.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 231795712 bytes 2026-02-22T19:56:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 286336000 bytes 2026-02-22T19:56:24.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 305861632 bytes 2026-02-22T19:56:25.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 348261376 bytes 2026-02-22T19:56:26.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 367758336 bytes 2026-02-22T19:56:26.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 375102464 bytes 2026-02-22T19:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 386709504 bytes 2026-02-22T19:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 484516864 bytes 2026-02-22T19:56:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T19:56:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T19:56:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T19:56:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:56:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:56:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:56:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:27.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:56:28.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T19:56:28.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:56:28.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T19:56:28.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":8903184.43,"entries_behind_primary":0,"entries_per_second":557.9,"non_primary_position":{"entry_tid":14533,"object_number":81,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":69,"tag_tid":4}} 2026-02-22T19:56:28.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.0 on trial066 2026-02-22T19:56:28.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:56:03 2026-02-22T19:56:28.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T19:56:28.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T19:56:28.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T19:56:28.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T19:56:28.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:56:03' 2026-02-22T19:56:28.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:56:28.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T19:56:28.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T19:56:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":8903184.43,"entries_behind_primary":0,"entries_per_second":557.9,"non_primary_position":{"entry_tid":14533,"object_number":81,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":69,"tag_tid":4}} 2026-02-22T19:56:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:56:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:56:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T19:56:28.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:56:28.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:28.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.0 on trial066 2026-02-22T19:56:28.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T19:56:28.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:56:03' 2026-02-22T19:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:56:03' 2026-02-22T19:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 19:56:03' 2026-02-22T19:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:56:28.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:28.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142024257mirror.0trial06619.2.3-689-gf90f4eaetrueOK42104263mirror.2trial06619.2.3-689-gf90f4eaefalseOK42134266mirror.3trial06619.2.3-689-gf90f4eaefalseOK43154318mirror.1trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":8903184.43,"entries_behind_primary":0,"entries_per_second":557.9,"non_primary_position":{"entry_tid":14533,"object_number":81,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":69,"tag_tid":4}}42024257mirror.0trial0662026-02-22 19:56:03cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 19:56:03' 2026-02-22T19:56:28.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T19:56:28.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:28.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:56:03' 2026-02-22T19:56:28.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T19:56:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:56:28.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T19:56:28.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:56:28.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:28.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:56:28.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap3 2026-02-22T19:56:28.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap3 2026-02-22T19:56:28.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:56:28.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:28.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:28.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap3 2026-02-22T19:56:28.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2026-02-22T19:56:59.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:56:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:59.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:57:00.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T19:57:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:57:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T19:57:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T19:57:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T19:57:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:57:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:57:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:57:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:57:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:57:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:57:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:00.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:00.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:00.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:00.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:00.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T19:57:00.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:57:00.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:57:00.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T19:57:00.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T19:57:00.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T19:57:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T19:57:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:57:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:57:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:57:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T19:57:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:57:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:57:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:57:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T19:57:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T19:57:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:00.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:00.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:00.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:00.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:00.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:00.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:00.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:00.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:00.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:00.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:00.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:00.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:00.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:00.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:00.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:00.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:00.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:00.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=22876], [object_number=111, tag_tid=8, entry_tid=22875], [object_number=110, tag_tid=8, entry_tid=22874], [object_number=109, tag_tid=8, entry_tid=22873]]], state=connected] 2026-02-22T19:57:00.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:00.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:00.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:00.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:00.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:00.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:00.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:00.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=22876], [object_number=111, tag_tid=8, entry_tid=22875], [object_number=110, tag_tid=8, entry_tid=22874], [object_number=109, tag_tid=8, entry_tid=22873]]], state=connected] 2026-02-22T19:57:01.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=108, tag_tid=8, entry_tid=22876' 2026-02-22T19:57:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=108, tag_tid=8, entry_tid=22876' 2026-02-22T19:57:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=22876' '!=' '' 2026-02-22T19:57:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=22876' = '' 2026-02-22T19:57:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=108, tag_tid=8, entry_tid=22876' 2026-02-22T19:57:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:01.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=22876' 2026-02-22T19:57:01.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:01.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:01.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:01.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:01.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=22876' 2026-02-22T19:57:01.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:01.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22876 2026-02-22T19:57:01.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 22876 2026-02-22T19:57:01.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:01.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:01.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:01.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:01.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:01.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:01.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:01.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:01.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:01.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:01.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:01.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:01.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:01.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:01.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=23128], [object_number=111, tag_tid=8, entry_tid=23127], [object_number=110, tag_tid=8, entry_tid=23126], [object_number=109, tag_tid=8, entry_tid=23125]]], state=connected] 2026-02-22T19:57:01.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:01.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:01.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:01.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=23128], [object_number=111, tag_tid=8, entry_tid=23127], [object_number=110, tag_tid=8, entry_tid=23126], [object_number=109, tag_tid=8, entry_tid=23125]]], state=connected] 2026-02-22T19:57:01.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=108, tag_tid=8, entry_tid=23128' 2026-02-22T19:57:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=108, tag_tid=8, entry_tid=23128' 2026-02-22T19:57:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=23128' '!=' 'object_number=108, tag_tid=8, entry_tid=22876' 2026-02-22T19:57:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=23128' = 'object_number=108, tag_tid=8, entry_tid=22876' 2026-02-22T19:57:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=108, tag_tid=8, entry_tid=23128' 2026-02-22T19:57:01.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:01.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:01.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=23128' 2026-02-22T19:57:01.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:01.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:01.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:01.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:01.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=23128' 2026-02-22T19:57:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23128 2026-02-22T19:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 23128 2026-02-22T19:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:01.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:01.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:01.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:01.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:01.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:01.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:01.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:01.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:01.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:01.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:01.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:01.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:01.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:01.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:01.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:01.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:01.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:01.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:01.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:01.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:01.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:01.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:01.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=110, tag_tid=8, entry_tid=23346], [object_number=109, tag_tid=8, entry_tid=23345], [object_number=108, tag_tid=8, entry_tid=23344], [object_number=111, tag_tid=8, entry_tid=23343]]], state=connected] 2026-02-22T19:57:01.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:01.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:01.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:01.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:01.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:01.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:01.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:01.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:01.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:01.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:01.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:01.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:01.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:01.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:01.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:01.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:01.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:01.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:01.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=110, tag_tid=8, entry_tid=23346], [object_number=109, tag_tid=8, entry_tid=23345], [object_number=108, tag_tid=8, entry_tid=23344], [object_number=111, tag_tid=8, entry_tid=23343]]], state=connected] 2026-02-22T19:57:01.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=110, tag_tid=8, entry_tid=23346' 2026-02-22T19:57:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=110, tag_tid=8, entry_tid=23346' 2026-02-22T19:57:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=110, tag_tid=8, entry_tid=23346' '!=' 'object_number=108, tag_tid=8, entry_tid=23128' 2026-02-22T19:57:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=110, tag_tid=8, entry_tid=23346' = 'object_number=108, tag_tid=8, entry_tid=23128' 2026-02-22T19:57:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=110, tag_tid=8, entry_tid=23346' 2026-02-22T19:57:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:01.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=110, tag_tid=8, entry_tid=23346' 2026-02-22T19:57:01.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:01.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:01.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=110, tag_tid=8, entry_tid=23346' 2026-02-22T19:57:01.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:01.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:01.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23346 2026-02-22T19:57:01.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 23346 2026-02-22T19:57:01.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:01.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:01.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:01.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:01.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:01.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:01.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:01.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:01.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:01.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:01.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:01.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:01.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:01.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:01.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:01.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:01.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:01.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:01.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:02.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:02.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:02.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:02.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:02.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:02.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=110, tag_tid=8, entry_tid=23586], [object_number=109, tag_tid=8, entry_tid=23585], [object_number=108, tag_tid=8, entry_tid=23584], [object_number=111, tag_tid=8, entry_tid=23583]]], state=connected] 2026-02-22T19:57:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:02.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:02.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:02.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:02.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=110, tag_tid=8, entry_tid=23586], [object_number=109, tag_tid=8, entry_tid=23585], [object_number=108, tag_tid=8, entry_tid=23584], [object_number=111, tag_tid=8, entry_tid=23583]]], state=connected] 2026-02-22T19:57:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=110, tag_tid=8, entry_tid=23586' 2026-02-22T19:57:02.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=110, tag_tid=8, entry_tid=23586' 2026-02-22T19:57:02.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=110, tag_tid=8, entry_tid=23586' '!=' 'object_number=110, tag_tid=8, entry_tid=23346' 2026-02-22T19:57:02.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:02.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=110, tag_tid=8, entry_tid=23586' = 'object_number=110, tag_tid=8, entry_tid=23346' 2026-02-22T19:57:02.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=110, tag_tid=8, entry_tid=23586' 2026-02-22T19:57:02.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:02.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:02.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:02.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=110, tag_tid=8, entry_tid=23586' 2026-02-22T19:57:02.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:02.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:02.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:02.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:02.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:02.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=110, tag_tid=8, entry_tid=23586' 2026-02-22T19:57:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23586 2026-02-22T19:57:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 23586 2026-02-22T19:57:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:02.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:02.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:02.274 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8664332948902944) > weight (0.25) 2026-02-22T19:57:02.274 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5314173821381805) > weight (0.25) 2026-02-22T19:57:02.274 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2026-02-22T19:57:02.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2026-02-22T19:57:02.274 INFO:tasks.rbd_mirror_thrash:waiting for 7.150349640994781 secs before reviving daemons 2026-02-22T19:57:02.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T19:57:02.268+0000 7f6d190bd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 20006) UID: 1000 2026-02-22T19:57:02.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:02.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:02.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:02.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:02.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:02.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:02.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:02.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:02.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:02.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:02.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:02.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:02.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:02.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:02.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:02.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:02.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:02.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:02.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:02.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:02.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:02.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:02.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:02.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:02.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:02.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:02.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:02.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:02.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=23816], [object_number=111, tag_tid=8, entry_tid=23815], [object_number=110, tag_tid=8, entry_tid=23814], [object_number=109, tag_tid=8, entry_tid=23813]]], state=connected] 2026-02-22T19:57:02.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:02.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:02.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:02.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:02.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=23816], [object_number=111, tag_tid=8, entry_tid=23815], [object_number=110, tag_tid=8, entry_tid=23814], [object_number=109, tag_tid=8, entry_tid=23813]]], state=connected] 2026-02-22T19:57:02.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=108, tag_tid=8, entry_tid=23816' 2026-02-22T19:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=108, tag_tid=8, entry_tid=23816' 2026-02-22T19:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=23816' '!=' 'object_number=110, tag_tid=8, entry_tid=23586' 2026-02-22T19:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=23816' = 'object_number=110, tag_tid=8, entry_tid=23586' 2026-02-22T19:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=108, tag_tid=8, entry_tid=23816' 2026-02-22T19:57:02.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:02.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:02.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=23816' 2026-02-22T19:57:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:02.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=23816' 2026-02-22T19:57:02.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:02.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23816 2026-02-22T19:57:02.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 23816 2026-02-22T19:57:02.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:02.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:02.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:02.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:02.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:02.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:02.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:02.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:02.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:02.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:02.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:02.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:02.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:02.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:02.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:02.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:02.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:02.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:02.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:02.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=24032], [object_number=111, tag_tid=8, entry_tid=24031], [object_number=110, tag_tid=8, entry_tid=24030], [object_number=109, tag_tid=8, entry_tid=24029]]], state=connected] 2026-02-22T19:57:02.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:02.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:02.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=24032], [object_number=111, tag_tid=8, entry_tid=24031], [object_number=110, tag_tid=8, entry_tid=24030], [object_number=109, tag_tid=8, entry_tid=24029]]], state=connected] 2026-02-22T19:57:02.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=108, tag_tid=8, entry_tid=24032' 2026-02-22T19:57:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=108, tag_tid=8, entry_tid=24032' 2026-02-22T19:57:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=24032' '!=' 'object_number=108, tag_tid=8, entry_tid=23816' 2026-02-22T19:57:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=24032' = 'object_number=108, tag_tid=8, entry_tid=23816' 2026-02-22T19:57:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=108, tag_tid=8, entry_tid=24032' 2026-02-22T19:57:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:02.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=24032' 2026-02-22T19:57:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:02.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=24032' 2026-02-22T19:57:02.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:02.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24032 2026-02-22T19:57:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 24032 2026-02-22T19:57:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:03.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:03.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:03.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:03.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:03.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:03.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:03.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:03.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:03.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=24272], [object_number=111, tag_tid=8, entry_tid=24271], [object_number=110, tag_tid=8, entry_tid=24270], [object_number=109, tag_tid=8, entry_tid=24269]]], state=connected] 2026-02-22T19:57:03.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:03.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:03.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:03.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:03.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:03.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:03.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:03.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=24272], [object_number=111, tag_tid=8, entry_tid=24271], [object_number=110, tag_tid=8, entry_tid=24270], [object_number=109, tag_tid=8, entry_tid=24269]]], state=connected] 2026-02-22T19:57:03.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=108, tag_tid=8, entry_tid=24272' 2026-02-22T19:57:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=108, tag_tid=8, entry_tid=24272' 2026-02-22T19:57:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=24272' '!=' 'object_number=108, tag_tid=8, entry_tid=24032' 2026-02-22T19:57:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=24272' = 'object_number=108, tag_tid=8, entry_tid=24032' 2026-02-22T19:57:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=108, tag_tid=8, entry_tid=24272' 2026-02-22T19:57:03.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:03.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:03.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=24272' 2026-02-22T19:57:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:03.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:03.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:03.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:03.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=24272' 2026-02-22T19:57:03.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:03.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24272 2026-02-22T19:57:03.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 24272 2026-02-22T19:57:03.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:03.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:03.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:03.418 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T19:57:03.419+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e096680 2026-02-22T19:57:03.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:03.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:03.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:03.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:03.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:03.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:03.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:03.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:03.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:03.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:03.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:03.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:03.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:03.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:03.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=109, tag_tid=8, entry_tid=24513], [object_number=108, tag_tid=8, entry_tid=24512], [object_number=111, tag_tid=8, entry_tid=24511], [object_number=110, tag_tid=8, entry_tid=24510]]], state=connected] 2026-02-22T19:57:03.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:03.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:03.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:03.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:03.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:03.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:03.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:03.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:03.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:03.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:03.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:03.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:03.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=109, tag_tid=8, entry_tid=24513], [object_number=108, tag_tid=8, entry_tid=24512], [object_number=111, tag_tid=8, entry_tid=24511], [object_number=110, tag_tid=8, entry_tid=24510]]], state=connected] 2026-02-22T19:57:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=109, tag_tid=8, entry_tid=24513' 2026-02-22T19:57:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=109, tag_tid=8, entry_tid=24513' 2026-02-22T19:57:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=109, tag_tid=8, entry_tid=24513' '!=' 'object_number=108, tag_tid=8, entry_tid=24272' 2026-02-22T19:57:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=109, tag_tid=8, entry_tid=24513' = 'object_number=108, tag_tid=8, entry_tid=24272' 2026-02-22T19:57:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=109, tag_tid=8, entry_tid=24513' 2026-02-22T19:57:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=109, tag_tid=8, entry_tid=24513' 2026-02-22T19:57:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:03.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:03.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:03.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:03.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=109, tag_tid=8, entry_tid=24513' 2026-02-22T19:57:03.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:03.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24513 2026-02-22T19:57:03.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 24513 2026-02-22T19:57:03.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:03.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:03.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:03.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:03.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:03.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:03.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:03.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:03.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=111, tag_tid=8, entry_tid=24727], [object_number=110, tag_tid=8, entry_tid=24726], [object_number=109, tag_tid=8, entry_tid=24725], [object_number=108, tag_tid=8, entry_tid=24724]]], state=connected] 2026-02-22T19:57:03.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:03.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:03.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:03.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:03.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:03.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:03.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:03.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:03.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:03.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:03.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:03.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:03.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:03.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:03.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:03.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=111, tag_tid=8, entry_tid=24727], [object_number=110, tag_tid=8, entry_tid=24726], [object_number=109, tag_tid=8, entry_tid=24725], [object_number=108, tag_tid=8, entry_tid=24724]]], state=connected] 2026-02-22T19:57:03.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:03.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=111, tag_tid=8, entry_tid=24727' 2026-02-22T19:57:03.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=111, tag_tid=8, entry_tid=24727' 2026-02-22T19:57:03.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=111, tag_tid=8, entry_tid=24727' '!=' 'object_number=109, tag_tid=8, entry_tid=24513' 2026-02-22T19:57:03.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:03.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=111, tag_tid=8, entry_tid=24727' = 'object_number=109, tag_tid=8, entry_tid=24513' 2026-02-22T19:57:03.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=111, tag_tid=8, entry_tid=24727' 2026-02-22T19:57:03.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:03.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:03.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:03.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=111, tag_tid=8, entry_tid=24727' 2026-02-22T19:57:03.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:03.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:03.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:03.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:03.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:03.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=111, tag_tid=8, entry_tid=24727' 2026-02-22T19:57:03.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:03.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24727 2026-02-22T19:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 24727 2026-02-22T19:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:03.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:04.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:04.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:04.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:04.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:04.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:04.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:04.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:04.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:04.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:04.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:04.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:04.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:04.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:04.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:04.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:04.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:04.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:04.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:04.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=24936], [object_number=111, tag_tid=8, entry_tid=24935], [object_number=110, tag_tid=8, entry_tid=24934], [object_number=109, tag_tid=8, entry_tid=24933]]], state=connected] 2026-02-22T19:57:04.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:04.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:04.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:04.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:04.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:04.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:04.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=24936], [object_number=111, tag_tid=8, entry_tid=24935], [object_number=110, tag_tid=8, entry_tid=24934], [object_number=109, tag_tid=8, entry_tid=24933]]], state=connected] 2026-02-22T19:57:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=108, tag_tid=8, entry_tid=24936' 2026-02-22T19:57:04.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=108, tag_tid=8, entry_tid=24936' 2026-02-22T19:57:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=24936' '!=' 'object_number=111, tag_tid=8, entry_tid=24727' 2026-02-22T19:57:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=24936' = 'object_number=111, tag_tid=8, entry_tid=24727' 2026-02-22T19:57:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=108, tag_tid=8, entry_tid=24936' 2026-02-22T19:57:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:04.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:04.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:04.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:04.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=24936' 2026-02-22T19:57:04.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:04.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:04.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:04.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:04.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:04.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=24936' 2026-02-22T19:57:04.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:04.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:04.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24936 2026-02-22T19:57:04.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 24936 2026-02-22T19:57:04.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:04.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:04.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:04.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:04.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:04.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:04.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:04.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:04.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:04.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:04.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:04.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:04.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:04.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:04.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:04.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:04.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:04.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:04.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:04.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:04.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:04.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:04.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:04.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:04.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:04.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:04.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=25156], [object_number=111, tag_tid=8, entry_tid=25155], [object_number=110, tag_tid=8, entry_tid=25154], [object_number=109, tag_tid=8, entry_tid=25153]]], state=connected] 2026-02-22T19:57:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:04.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:04.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:04.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:04.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:04.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 27 2026-02-22T19:57:04.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:04.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:04.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:04.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=108, tag_tid=8, entry_tid=25156], [object_number=111, tag_tid=8, entry_tid=25155], [object_number=110, tag_tid=8, entry_tid=25154], [object_number=109, tag_tid=8, entry_tid=25153]]], state=connected] 2026-02-22T19:57:04.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=108, tag_tid=8, entry_tid=25156' 2026-02-22T19:57:04.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=108, tag_tid=8, entry_tid=25156' 2026-02-22T19:57:04.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=25156' '!=' 'object_number=108, tag_tid=8, entry_tid=24936' 2026-02-22T19:57:04.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:04.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=108, tag_tid=8, entry_tid=25156' = 'object_number=108, tag_tid=8, entry_tid=24936' 2026-02-22T19:57:04.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=108, tag_tid=8, entry_tid=25156' 2026-02-22T19:57:04.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:04.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:04.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:04.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:04.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=25156' 2026-02-22T19:57:04.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:04.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:04.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:04.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:04.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:04.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:04.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=108, tag_tid=8, entry_tid=25156' 2026-02-22T19:57:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:04.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:04.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25156 2026-02-22T19:57:04.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 25156 2026-02-22T19:57:04.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:04.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:04.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:04.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:04.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:04.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:04.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:04.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:04.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:04.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:04.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:04.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:04.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:04.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:04.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:04.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:04.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:04.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:04.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:05.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:05.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:05.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:05.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=25291], [object_number=114, tag_tid=8, entry_tid=25290], [object_number=113, tag_tid=8, entry_tid=25289], [object_number=112, tag_tid=8, entry_tid=25288]]], state=connected] 2026-02-22T19:57:05.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:05.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:05.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:05.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:05.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:05.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:05.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:05.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:05.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=25291], [object_number=114, tag_tid=8, entry_tid=25290], [object_number=113, tag_tid=8, entry_tid=25289], [object_number=112, tag_tid=8, entry_tid=25288]]], state=connected] 2026-02-22T19:57:05.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=115, tag_tid=8, entry_tid=25291' 2026-02-22T19:57:05.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=115, tag_tid=8, entry_tid=25291' 2026-02-22T19:57:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=25291' '!=' 'object_number=108, tag_tid=8, entry_tid=25156' 2026-02-22T19:57:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=25291' = 'object_number=108, tag_tid=8, entry_tid=25156' 2026-02-22T19:57:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=115, tag_tid=8, entry_tid=25291' 2026-02-22T19:57:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:05.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=25291' 2026-02-22T19:57:05.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:05.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:05.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:05.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:05.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=25291' 2026-02-22T19:57:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25291 2026-02-22T19:57:05.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 25291 2026-02-22T19:57:05.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:05.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:05.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:05.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:05.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:05.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:05.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:05.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:05.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:05.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:05.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:05.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:05.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:05.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=25561], [object_number=112, tag_tid=8, entry_tid=25560], [object_number=115, tag_tid=8, entry_tid=25559], [object_number=114, tag_tid=8, entry_tid=25558]]], state=connected] 2026-02-22T19:57:05.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:05.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:05.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:05.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:05.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:05.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:05.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:05.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:05.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:05.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:05.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=25561], [object_number=112, tag_tid=8, entry_tid=25560], [object_number=115, tag_tid=8, entry_tid=25559], [object_number=114, tag_tid=8, entry_tid=25558]]], state=connected] 2026-02-22T19:57:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=113, tag_tid=8, entry_tid=25561' 2026-02-22T19:57:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=113, tag_tid=8, entry_tid=25561' 2026-02-22T19:57:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=25561' '!=' 'object_number=115, tag_tid=8, entry_tid=25291' 2026-02-22T19:57:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=25561' = 'object_number=115, tag_tid=8, entry_tid=25291' 2026-02-22T19:57:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=113, tag_tid=8, entry_tid=25561' 2026-02-22T19:57:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=25561' 2026-02-22T19:57:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:05.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:05.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:05.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:05.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=25561' 2026-02-22T19:57:05.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:05.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25561 2026-02-22T19:57:05.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 25561 2026-02-22T19:57:05.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:05.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:05.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:05.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:05.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:05.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:05.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:05.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:05.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:05.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:05.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:05.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:05.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:05.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:05.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=25761], [object_number=112, tag_tid=8, entry_tid=25760], [object_number=115, tag_tid=8, entry_tid=25759], [object_number=114, tag_tid=8, entry_tid=25758]]], state=connected] 2026-02-22T19:57:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:05.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:05.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:05.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:05.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:05.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:05.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:05.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:05.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=25761], [object_number=112, tag_tid=8, entry_tid=25760], [object_number=115, tag_tid=8, entry_tid=25759], [object_number=114, tag_tid=8, entry_tid=25758]]], state=connected] 2026-02-22T19:57:05.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=113, tag_tid=8, entry_tid=25761' 2026-02-22T19:57:05.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=113, tag_tid=8, entry_tid=25761' 2026-02-22T19:57:05.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=25761' '!=' 'object_number=113, tag_tid=8, entry_tid=25561' 2026-02-22T19:57:05.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:05.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=25761' = 'object_number=113, tag_tid=8, entry_tid=25561' 2026-02-22T19:57:05.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=113, tag_tid=8, entry_tid=25761' 2026-02-22T19:57:05.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:05.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:05.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:05.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=25761' 2026-02-22T19:57:05.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:05.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:05.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:05.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:05.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:05.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=25761' 2026-02-22T19:57:05.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:05.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:05.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25761 2026-02-22T19:57:05.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 25761 2026-02-22T19:57:05.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:05.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:05.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:06.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:06.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:06.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:06.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:06.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:06.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:06.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:06.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:06.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:06.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:06.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:06.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:06.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:06.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:06.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:06.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:06.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:06.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:06.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=25963], [object_number=114, tag_tid=8, entry_tid=25962], [object_number=113, tag_tid=8, entry_tid=25961], [object_number=112, tag_tid=8, entry_tid=25960]]], state=connected] 2026-02-22T19:57:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:06.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=25963], [object_number=114, tag_tid=8, entry_tid=25962], [object_number=113, tag_tid=8, entry_tid=25961], [object_number=112, tag_tid=8, entry_tid=25960]]], state=connected] 2026-02-22T19:57:06.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=115, tag_tid=8, entry_tid=25963' 2026-02-22T19:57:06.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=115, tag_tid=8, entry_tid=25963' 2026-02-22T19:57:06.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=25963' '!=' 'object_number=113, tag_tid=8, entry_tid=25761' 2026-02-22T19:57:06.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:06.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=25963' = 'object_number=113, tag_tid=8, entry_tid=25761' 2026-02-22T19:57:06.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=115, tag_tid=8, entry_tid=25963' 2026-02-22T19:57:06.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:06.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:06.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:06.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=25963' 2026-02-22T19:57:06.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:06.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:06.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:06.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:06.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:06.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=25963' 2026-02-22T19:57:06.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:06.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25963 2026-02-22T19:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 25963 2026-02-22T19:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:06.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:06.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:06.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:06.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:06.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:06.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:06.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:06.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:06.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:06.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:06.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:06.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:06.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:06.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:06.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:06.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:06.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:06.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:06.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:06.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:06.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:06.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:06.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:06.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:06.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:06.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:06.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:06.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:06.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:06.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:06.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:06.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=26187], [object_number=114, tag_tid=8, entry_tid=26186], [object_number=113, tag_tid=8, entry_tid=26185], [object_number=112, tag_tid=8, entry_tid=26184]]], state=connected] 2026-02-22T19:57:06.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:06.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:06.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:06.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:06.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:06.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:06.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:06.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:06.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=26187], [object_number=114, tag_tid=8, entry_tid=26186], [object_number=113, tag_tid=8, entry_tid=26185], [object_number=112, tag_tid=8, entry_tid=26184]]], state=connected] 2026-02-22T19:57:06.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=115, tag_tid=8, entry_tid=26187' 2026-02-22T19:57:06.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=115, tag_tid=8, entry_tid=26187' 2026-02-22T19:57:06.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=26187' '!=' 'object_number=115, tag_tid=8, entry_tid=25963' 2026-02-22T19:57:06.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:06.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=26187' = 'object_number=115, tag_tid=8, entry_tid=25963' 2026-02-22T19:57:06.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=115, tag_tid=8, entry_tid=26187' 2026-02-22T19:57:06.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:06.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:06.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:06.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=26187' 2026-02-22T19:57:06.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:06.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:06.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:06.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:06.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:06.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=26187' 2026-02-22T19:57:06.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:06.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26187 2026-02-22T19:57:06.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 26187 2026-02-22T19:57:06.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:06.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:06.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:06.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:06.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:06.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:06.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:06.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:06.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:06.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:06.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:06.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:06.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:06.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:06.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:06.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:06.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:06.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:06.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:06.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:06.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:06.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=26417], [object_number=112, tag_tid=8, entry_tid=26416], [object_number=115, tag_tid=8, entry_tid=26415], [object_number=114, tag_tid=8, entry_tid=26414]]], state=connected] 2026-02-22T19:57:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:06.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:06.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:06.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:06.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:06.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:06.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:06.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:06.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:06.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:06.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:06.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=26417], [object_number=112, tag_tid=8, entry_tid=26416], [object_number=115, tag_tid=8, entry_tid=26415], [object_number=114, tag_tid=8, entry_tid=26414]]], state=connected] 2026-02-22T19:57:06.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:06.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=113, tag_tid=8, entry_tid=26417' 2026-02-22T19:57:06.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=113, tag_tid=8, entry_tid=26417' 2026-02-22T19:57:06.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=26417' '!=' 'object_number=115, tag_tid=8, entry_tid=26187' 2026-02-22T19:57:06.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:06.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=26417' = 'object_number=115, tag_tid=8, entry_tid=26187' 2026-02-22T19:57:06.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=113, tag_tid=8, entry_tid=26417' 2026-02-22T19:57:06.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:06.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:06.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:06.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=26417' 2026-02-22T19:57:06.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:06.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:06.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:06.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:06.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=26417' 2026-02-22T19:57:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26417 2026-02-22T19:57:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 26417 2026-02-22T19:57:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:07.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:07.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:07.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:07.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:07.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:07.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:07.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:07.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:07.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:07.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:07.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:07.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:07.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:07.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:07.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:07.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:07.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:07.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:07.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:07.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:07.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:07.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:07.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:07.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:07.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:07.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:07.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:07.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:07.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:07.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=114, tag_tid=8, entry_tid=26666], [object_number=113, tag_tid=8, entry_tid=26665], [object_number=112, tag_tid=8, entry_tid=26664], [object_number=115, tag_tid=8, entry_tid=26663]]], state=connected] 2026-02-22T19:57:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:07.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:07.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:07.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:07.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:07.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:07.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:07.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:07.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:07.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:07.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:07.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:07.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=114, tag_tid=8, entry_tid=26666], [object_number=113, tag_tid=8, entry_tid=26665], [object_number=112, tag_tid=8, entry_tid=26664], [object_number=115, tag_tid=8, entry_tid=26663]]], state=connected] 2026-02-22T19:57:07.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=114, tag_tid=8, entry_tid=26666' 2026-02-22T19:57:07.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=114, tag_tid=8, entry_tid=26666' 2026-02-22T19:57:07.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=114, tag_tid=8, entry_tid=26666' '!=' 'object_number=113, tag_tid=8, entry_tid=26417' 2026-02-22T19:57:07.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:07.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=114, tag_tid=8, entry_tid=26666' = 'object_number=113, tag_tid=8, entry_tid=26417' 2026-02-22T19:57:07.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=114, tag_tid=8, entry_tid=26666' 2026-02-22T19:57:07.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:07.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:07.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:07.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=114, tag_tid=8, entry_tid=26666' 2026-02-22T19:57:07.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:07.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:07.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=114, tag_tid=8, entry_tid=26666' 2026-02-22T19:57:07.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:07.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26666 2026-02-22T19:57:07.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 26666 2026-02-22T19:57:07.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:07.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:07.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:07.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:07.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:07.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:07.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:07.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:07.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:07.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:07.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:07.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:07.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:07.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:07.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:07.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:07.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:07.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:07.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:07.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:07.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:07.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:07.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:07.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:07.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:07.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:07.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:07.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=112, tag_tid=8, entry_tid=26896], [object_number=115, tag_tid=8, entry_tid=26895], [object_number=114, tag_tid=8, entry_tid=26894], [object_number=113, tag_tid=8, entry_tid=26893]]], state=connected] 2026-02-22T19:57:07.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:07.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:07.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:07.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:07.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:07.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:07.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:07.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:07.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=112, tag_tid=8, entry_tid=26896], [object_number=115, tag_tid=8, entry_tid=26895], [object_number=114, tag_tid=8, entry_tid=26894], [object_number=113, tag_tid=8, entry_tid=26893]]], state=connected] 2026-02-22T19:57:07.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=112, tag_tid=8, entry_tid=26896' 2026-02-22T19:57:07.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=112, tag_tid=8, entry_tid=26896' 2026-02-22T19:57:07.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=112, tag_tid=8, entry_tid=26896' '!=' 'object_number=114, tag_tid=8, entry_tid=26666' 2026-02-22T19:57:07.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=112, tag_tid=8, entry_tid=26896' = 'object_number=114, tag_tid=8, entry_tid=26666' 2026-02-22T19:57:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=112, tag_tid=8, entry_tid=26896' 2026-02-22T19:57:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:07.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=112, tag_tid=8, entry_tid=26896' 2026-02-22T19:57:07.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:07.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:07.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=112, tag_tid=8, entry_tid=26896' 2026-02-22T19:57:07.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:07.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26896 2026-02-22T19:57:07.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 26896 2026-02-22T19:57:07.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:07.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:07.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:07.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:07.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:07.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:07.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:07.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:07.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:07.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:07.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:07.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:07.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:07.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:07.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:07.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:07.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=27163], [object_number=114, tag_tid=8, entry_tid=27162], [object_number=113, tag_tid=8, entry_tid=27161], [object_number=112, tag_tid=8, entry_tid=27160]]], state=connected] 2026-02-22T19:57:07.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:07.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:07.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:07.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:07.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:07.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:07.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:07.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:07.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:07.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:07.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:07.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=27163], [object_number=114, tag_tid=8, entry_tid=27162], [object_number=113, tag_tid=8, entry_tid=27161], [object_number=112, tag_tid=8, entry_tid=27160]]], state=connected] 2026-02-22T19:57:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:07.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=115, tag_tid=8, entry_tid=27163' 2026-02-22T19:57:07.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=115, tag_tid=8, entry_tid=27163' 2026-02-22T19:57:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=27163' '!=' 'object_number=112, tag_tid=8, entry_tid=26896' 2026-02-22T19:57:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=27163' = 'object_number=112, tag_tid=8, entry_tid=26896' 2026-02-22T19:57:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=115, tag_tid=8, entry_tid=27163' 2026-02-22T19:57:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=27163' 2026-02-22T19:57:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:07.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:07.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:07.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:07.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=27163' 2026-02-22T19:57:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27163 2026-02-22T19:57:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 27163 2026-02-22T19:57:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:08.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:08.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:08.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:08.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:08.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:08.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:08.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:08.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:08.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:08.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:08.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:08.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:08.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:08.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:08.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:08.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:08.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:08.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:08.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:08.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=114, tag_tid=8, entry_tid=27394], [object_number=113, tag_tid=8, entry_tid=27393], [object_number=112, tag_tid=8, entry_tid=27392], [object_number=115, tag_tid=8, entry_tid=27391]]], state=connected] 2026-02-22T19:57:08.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:08.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:08.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:08.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:08.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:08.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:08.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:08.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:08.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=114, tag_tid=8, entry_tid=27394], [object_number=113, tag_tid=8, entry_tid=27393], [object_number=112, tag_tid=8, entry_tid=27392], [object_number=115, tag_tid=8, entry_tid=27391]]], state=connected] 2026-02-22T19:57:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=114, tag_tid=8, entry_tid=27394' 2026-02-22T19:57:08.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=114, tag_tid=8, entry_tid=27394' 2026-02-22T19:57:08.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=114, tag_tid=8, entry_tid=27394' '!=' 'object_number=115, tag_tid=8, entry_tid=27163' 2026-02-22T19:57:08.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=114, tag_tid=8, entry_tid=27394' = 'object_number=115, tag_tid=8, entry_tid=27163' 2026-02-22T19:57:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=114, tag_tid=8, entry_tid=27394' 2026-02-22T19:57:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=114, tag_tid=8, entry_tid=27394' 2026-02-22T19:57:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:08.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=114, tag_tid=8, entry_tid=27394' 2026-02-22T19:57:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:08.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27394 2026-02-22T19:57:08.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 27394 2026-02-22T19:57:08.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:08.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:08.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:08.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:08.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:08.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:08.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:08.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:08.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:08.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:08.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:08.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:08.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:08.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:08.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:08.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:08.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:08.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:08.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:08.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:08.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:08.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:08.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:08.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:08.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:08.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=27627], [object_number=114, tag_tid=8, entry_tid=27626], [object_number=113, tag_tid=8, entry_tid=27625], [object_number=112, tag_tid=8, entry_tid=27624]]], state=connected] 2026-02-22T19:57:08.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:08.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:08.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:08.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:08.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:08.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:08.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:08.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:08.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:08.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:08.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=27627], [object_number=114, tag_tid=8, entry_tid=27626], [object_number=113, tag_tid=8, entry_tid=27625], [object_number=112, tag_tid=8, entry_tid=27624]]], state=connected] 2026-02-22T19:57:08.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:08.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=115, tag_tid=8, entry_tid=27627' 2026-02-22T19:57:08.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=115, tag_tid=8, entry_tid=27627' 2026-02-22T19:57:08.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=27627' '!=' 'object_number=114, tag_tid=8, entry_tid=27394' 2026-02-22T19:57:08.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:08.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=27627' = 'object_number=114, tag_tid=8, entry_tid=27394' 2026-02-22T19:57:08.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=115, tag_tid=8, entry_tid=27627' 2026-02-22T19:57:08.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:08.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:08.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:08.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=27627' 2026-02-22T19:57:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:08.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:08.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=27627' 2026-02-22T19:57:08.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:08.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:08.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27627 2026-02-22T19:57:08.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 27627 2026-02-22T19:57:08.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:08.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:08.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:08.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:08.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:08.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:08.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:08.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:09.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:09.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:09.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:09.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:09.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:09.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:09.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=27895], [object_number=114, tag_tid=8, entry_tid=27894], [object_number=113, tag_tid=8, entry_tid=27893], [object_number=112, tag_tid=8, entry_tid=27892]]], state=connected] 2026-02-22T19:57:09.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:09.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:09.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:09.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:09.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:09.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:09.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:09.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:09.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:09.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=27895], [object_number=114, tag_tid=8, entry_tid=27894], [object_number=113, tag_tid=8, entry_tid=27893], [object_number=112, tag_tid=8, entry_tid=27892]]], state=connected] 2026-02-22T19:57:09.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=115, tag_tid=8, entry_tid=27895' 2026-02-22T19:57:09.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=115, tag_tid=8, entry_tid=27895' 2026-02-22T19:57:09.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=27895' '!=' 'object_number=115, tag_tid=8, entry_tid=27627' 2026-02-22T19:57:09.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:09.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=27895' = 'object_number=115, tag_tid=8, entry_tid=27627' 2026-02-22T19:57:09.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=115, tag_tid=8, entry_tid=27895' 2026-02-22T19:57:09.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:09.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:09.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:09.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=27895' 2026-02-22T19:57:09.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:09.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:09.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=27895' 2026-02-22T19:57:09.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:09.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27895 2026-02-22T19:57:09.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 27895 2026-02-22T19:57:09.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:09.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:09.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:09.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:09.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:09.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:09.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:09.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:09.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:09.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:09.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:09.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:09.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:09.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:09.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:09.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:09.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=28111], [object_number=114, tag_tid=8, entry_tid=28110], [object_number=113, tag_tid=8, entry_tid=28109], [object_number=112, tag_tid=8, entry_tid=28108]]], state=connected] 2026-02-22T19:57:09.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:09.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:09.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.426 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2026-02-22T19:57:09.426 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T19:57:09.426 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2026-02-22T19:57:09.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2026-02-22T19:57:09.427 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2026-02-22T19:57:09.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2026-02-22T19:57:09.428 INFO:tasks.rbd_mirror_thrash:waiting for 96.74882021341979 secs before thrashing 2026-02-22T19:57:09.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:09.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:09.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:09.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:09.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=28111], [object_number=114, tag_tid=8, entry_tid=28110], [object_number=113, tag_tid=8, entry_tid=28109], [object_number=112, tag_tid=8, entry_tid=28108]]], state=connected] 2026-02-22T19:57:09.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=115, tag_tid=8, entry_tid=28111' 2026-02-22T19:57:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=115, tag_tid=8, entry_tid=28111' 2026-02-22T19:57:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=28111' '!=' 'object_number=115, tag_tid=8, entry_tid=27895' 2026-02-22T19:57:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=28111' = 'object_number=115, tag_tid=8, entry_tid=27895' 2026-02-22T19:57:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=115, tag_tid=8, entry_tid=28111' 2026-02-22T19:57:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:09.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:09.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=28111' 2026-02-22T19:57:09.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:09.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:09.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=28111' 2026-02-22T19:57:09.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:09.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28111 2026-02-22T19:57:09.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 28111 2026-02-22T19:57:09.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:09.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:09.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:09.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:09.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:09.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:09.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:09.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:09.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:09.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:09.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=28199], [object_number=114, tag_tid=8, entry_tid=28198], [object_number=113, tag_tid=8, entry_tid=28197], [object_number=112, tag_tid=8, entry_tid=28196]]], state=connected] 2026-02-22T19:57:09.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:09.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:09.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:09.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:09.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:09.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:09.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:09.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:09.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=28199], [object_number=114, tag_tid=8, entry_tid=28198], [object_number=113, tag_tid=8, entry_tid=28197], [object_number=112, tag_tid=8, entry_tid=28196]]], state=connected] 2026-02-22T19:57:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:09.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=115, tag_tid=8, entry_tid=28199' 2026-02-22T19:57:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=115, tag_tid=8, entry_tid=28199' 2026-02-22T19:57:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=28199' '!=' 'object_number=115, tag_tid=8, entry_tid=28111' 2026-02-22T19:57:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=28199' = 'object_number=115, tag_tid=8, entry_tid=28111' 2026-02-22T19:57:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=115, tag_tid=8, entry_tid=28199' 2026-02-22T19:57:09.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:09.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:09.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=28199' 2026-02-22T19:57:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:09.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:09.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:09.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=28199' 2026-02-22T19:57:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28199 2026-02-22T19:57:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 28199 2026-02-22T19:57:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:10.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:10.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:10.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:10.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:10.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:10.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:10.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:10.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:10.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:10.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:10.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:10.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=28427], [object_number=114, tag_tid=8, entry_tid=28426], [object_number=113, tag_tid=8, entry_tid=28425], [object_number=112, tag_tid=8, entry_tid=28424]]], state=connected] 2026-02-22T19:57:10.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:10.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:10.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:10.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:10.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=115, tag_tid=8, entry_tid=28427], [object_number=114, tag_tid=8, entry_tid=28426], [object_number=113, tag_tid=8, entry_tid=28425], [object_number=112, tag_tid=8, entry_tid=28424]]], state=connected] 2026-02-22T19:57:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=115, tag_tid=8, entry_tid=28427' 2026-02-22T19:57:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=115, tag_tid=8, entry_tid=28427' 2026-02-22T19:57:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=28427' '!=' 'object_number=115, tag_tid=8, entry_tid=28199' 2026-02-22T19:57:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=115, tag_tid=8, entry_tid=28427' = 'object_number=115, tag_tid=8, entry_tid=28199' 2026-02-22T19:57:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=115, tag_tid=8, entry_tid=28427' 2026-02-22T19:57:10.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:10.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:10.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=28427' 2026-02-22T19:57:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:10.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:10.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:10.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:10.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:10.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:10.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:10.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=115, tag_tid=8, entry_tid=28427' 2026-02-22T19:57:10.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:10.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28427 2026-02-22T19:57:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 28427 2026-02-22T19:57:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:10.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:10.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:10.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:10.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:10.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:10.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:10.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:10.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:10.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:10.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:10.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:10.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:10.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:10.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:10.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:10.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:10.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:10.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:10.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:10.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:10.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:10.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:10.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:10.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:10.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:10.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:10.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=28797], [object_number=112, tag_tid=8, entry_tid=28796], [object_number=115, tag_tid=8, entry_tid=28795], [object_number=114, tag_tid=8, entry_tid=28794]]], state=connected] 2026-02-22T19:57:10.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:10.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:10.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:10.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:10.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:10.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:10.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:10.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:10.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:10.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:10.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:10.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:10.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:10.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:10.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:10.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:10.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:10.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:10.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=28797], [object_number=112, tag_tid=8, entry_tid=28796], [object_number=115, tag_tid=8, entry_tid=28795], [object_number=114, tag_tid=8, entry_tid=28794]]], state=connected] 2026-02-22T19:57:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:10.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=113, tag_tid=8, entry_tid=28797' 2026-02-22T19:57:10.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=113, tag_tid=8, entry_tid=28797' 2026-02-22T19:57:10.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=28797' '!=' 'object_number=115, tag_tid=8, entry_tid=28427' 2026-02-22T19:57:10.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:10.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=28797' = 'object_number=115, tag_tid=8, entry_tid=28427' 2026-02-22T19:57:10.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=113, tag_tid=8, entry_tid=28797' 2026-02-22T19:57:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:10.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=28797' 2026-02-22T19:57:10.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:10.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:10.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:10.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:10.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:10.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:10.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:10.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=28797' 2026-02-22T19:57:10.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:10.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28797 2026-02-22T19:57:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 28797 2026-02-22T19:57:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:10.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:10.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:10.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:10.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:10.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:10.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:10.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:10.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:10.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:10.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:10.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:10.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:10.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:10.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:10.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:10.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:10.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:11.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:11.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:11.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:11.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:11.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=29021], [object_number=112, tag_tid=8, entry_tid=29020], [object_number=115, tag_tid=8, entry_tid=29019], [object_number=114, tag_tid=8, entry_tid=29018]]], state=connected] 2026-02-22T19:57:11.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:11.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:11.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:11.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:11.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=29021], [object_number=112, tag_tid=8, entry_tid=29020], [object_number=115, tag_tid=8, entry_tid=29019], [object_number=114, tag_tid=8, entry_tid=29018]]], state=connected] 2026-02-22T19:57:11.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=113, tag_tid=8, entry_tid=29021' 2026-02-22T19:57:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=113, tag_tid=8, entry_tid=29021' 2026-02-22T19:57:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=29021' '!=' 'object_number=113, tag_tid=8, entry_tid=28797' 2026-02-22T19:57:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=29021' = 'object_number=113, tag_tid=8, entry_tid=28797' 2026-02-22T19:57:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=113, tag_tid=8, entry_tid=29021' 2026-02-22T19:57:11.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:11.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:11.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=29021' 2026-02-22T19:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=29021' 2026-02-22T19:57:11.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:11.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29021 2026-02-22T19:57:11.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 29021 2026-02-22T19:57:11.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:11.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:11.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:11.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:11.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:11.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:11.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:11.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:11.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:11.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:11.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:11.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:11.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:11.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:11.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:11.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:11.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:11.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:11.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:11.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:11.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:11.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:11.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:11.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:11.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:11.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:11.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=29253], [object_number=112, tag_tid=8, entry_tid=29252], [object_number=115, tag_tid=8, entry_tid=29251], [object_number=114, tag_tid=8, entry_tid=29250]]], state=connected] 2026-02-22T19:57:11.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:11.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:11.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:11.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:11.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:11.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:11.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:11.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:11.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:11.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:11.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:11.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:11.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=29253], [object_number=112, tag_tid=8, entry_tid=29252], [object_number=115, tag_tid=8, entry_tid=29251], [object_number=114, tag_tid=8, entry_tid=29250]]], state=connected] 2026-02-22T19:57:11.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=113, tag_tid=8, entry_tid=29253' 2026-02-22T19:57:11.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=113, tag_tid=8, entry_tid=29253' 2026-02-22T19:57:11.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=29253' '!=' 'object_number=113, tag_tid=8, entry_tid=29021' 2026-02-22T19:57:11.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:11.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=29253' = 'object_number=113, tag_tid=8, entry_tid=29021' 2026-02-22T19:57:11.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=113, tag_tid=8, entry_tid=29253' 2026-02-22T19:57:11.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:11.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:11.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:11.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=29253' 2026-02-22T19:57:11.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:11.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=29253' 2026-02-22T19:57:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29253 2026-02-22T19:57:11.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 29253 2026-02-22T19:57:11.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:11.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:11.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:11.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:11.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:11.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:11.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:11.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:11.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:11.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:11.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:11.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:11.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:11.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:11.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=29341], [object_number=112, tag_tid=8, entry_tid=29340], [object_number=115, tag_tid=8, entry_tid=29339], [object_number=114, tag_tid=8, entry_tid=29338]]], state=connected] 2026-02-22T19:57:11.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:11.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:11.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:11.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:11.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 28 2026-02-22T19:57:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=113, tag_tid=8, entry_tid=29341], [object_number=112, tag_tid=8, entry_tid=29340], [object_number=115, tag_tid=8, entry_tid=29339], [object_number=114, tag_tid=8, entry_tid=29338]]], state=connected] 2026-02-22T19:57:11.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:11.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=113, tag_tid=8, entry_tid=29341' 2026-02-22T19:57:11.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=113, tag_tid=8, entry_tid=29341' 2026-02-22T19:57:11.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=29341' '!=' 'object_number=113, tag_tid=8, entry_tid=29253' 2026-02-22T19:57:11.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:11.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=113, tag_tid=8, entry_tid=29341' = 'object_number=113, tag_tid=8, entry_tid=29253' 2026-02-22T19:57:11.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=113, tag_tid=8, entry_tid=29341' 2026-02-22T19:57:11.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:11.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:11.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:11.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=29341' 2026-02-22T19:57:11.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:11.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:11.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:11.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:11.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:11.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=113, tag_tid=8, entry_tid=29341' 2026-02-22T19:57:11.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:11.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29341 2026-02-22T19:57:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 29341 2026-02-22T19:57:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:12.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:12.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:12.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:12.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:12.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:12.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:12.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:12.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:12.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:12.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:12.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:12.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:12.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:12.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:12.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:12.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:12.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:12.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:12.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=116, tag_tid=8, entry_tid=29524], [object_number=119, tag_tid=8, entry_tid=29523], [object_number=118, tag_tid=8, entry_tid=29522], [object_number=117, tag_tid=8, entry_tid=29521]]], state=connected] 2026-02-22T19:57:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:12.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:12.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:12.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:12.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:12.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:12.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=116, tag_tid=8, entry_tid=29524], [object_number=119, tag_tid=8, entry_tid=29523], [object_number=118, tag_tid=8, entry_tid=29522], [object_number=117, tag_tid=8, entry_tid=29521]]], state=connected] 2026-02-22T19:57:12.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=116, tag_tid=8, entry_tid=29524' 2026-02-22T19:57:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=116, tag_tid=8, entry_tid=29524' 2026-02-22T19:57:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=116, tag_tid=8, entry_tid=29524' '!=' 'object_number=113, tag_tid=8, entry_tid=29341' 2026-02-22T19:57:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:12.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=116, tag_tid=8, entry_tid=29524' = 'object_number=113, tag_tid=8, entry_tid=29341' 2026-02-22T19:57:12.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=116, tag_tid=8, entry_tid=29524' 2026-02-22T19:57:12.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:12.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:12.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:12.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=116, tag_tid=8, entry_tid=29524' 2026-02-22T19:57:12.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:12.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:12.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=116, tag_tid=8, entry_tid=29524' 2026-02-22T19:57:12.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:12.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29524 2026-02-22T19:57:12.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 29524 2026-02-22T19:57:12.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:12.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:12.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:12.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:12.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:12.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:12.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:12.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:12.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:12.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:12.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:12.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:12.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:12.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:12.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:12.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:12.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:12.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=116, tag_tid=8, entry_tid=29752], [object_number=119, tag_tid=8, entry_tid=29751], [object_number=118, tag_tid=8, entry_tid=29750], [object_number=117, tag_tid=8, entry_tid=29749]]], state=connected] 2026-02-22T19:57:12.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:12.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:12.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:12.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:12.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:12.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:12.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:12.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:12.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:12.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:12.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:12.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:12.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=116, tag_tid=8, entry_tid=29752], [object_number=119, tag_tid=8, entry_tid=29751], [object_number=118, tag_tid=8, entry_tid=29750], [object_number=117, tag_tid=8, entry_tid=29749]]], state=connected] 2026-02-22T19:57:12.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=116, tag_tid=8, entry_tid=29752' 2026-02-22T19:57:12.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=116, tag_tid=8, entry_tid=29752' 2026-02-22T19:57:12.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=116, tag_tid=8, entry_tid=29752' '!=' 'object_number=116, tag_tid=8, entry_tid=29524' 2026-02-22T19:57:12.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:12.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=116, tag_tid=8, entry_tid=29752' = 'object_number=116, tag_tid=8, entry_tid=29524' 2026-02-22T19:57:12.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=116, tag_tid=8, entry_tid=29752' 2026-02-22T19:57:12.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:12.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:12.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:12.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=116, tag_tid=8, entry_tid=29752' 2026-02-22T19:57:12.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:12.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:12.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:12.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:12.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:12.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=116, tag_tid=8, entry_tid=29752' 2026-02-22T19:57:12.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:12.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29752 2026-02-22T19:57:12.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 29752 2026-02-22T19:57:12.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:12.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:12.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:12.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:12.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:12.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:12.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:12.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:12.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:12.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:12.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:12.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:12.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:12.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:12.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=119, tag_tid=8, entry_tid=29991], [object_number=118, tag_tid=8, entry_tid=29990], [object_number=117, tag_tid=8, entry_tid=29989], [object_number=116, tag_tid=8, entry_tid=29988]]], state=connected] 2026-02-22T19:57:12.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:12.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:12.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:12.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:12.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:12.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=119, tag_tid=8, entry_tid=29991], [object_number=118, tag_tid=8, entry_tid=29990], [object_number=117, tag_tid=8, entry_tid=29989], [object_number=116, tag_tid=8, entry_tid=29988]]], state=connected] 2026-02-22T19:57:12.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:12.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=119, tag_tid=8, entry_tid=29991' 2026-02-22T19:57:12.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=119, tag_tid=8, entry_tid=29991' 2026-02-22T19:57:12.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=119, tag_tid=8, entry_tid=29991' '!=' 'object_number=116, tag_tid=8, entry_tid=29752' 2026-02-22T19:57:12.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:12.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=119, tag_tid=8, entry_tid=29991' = 'object_number=116, tag_tid=8, entry_tid=29752' 2026-02-22T19:57:12.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=119, tag_tid=8, entry_tid=29991' 2026-02-22T19:57:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=119, tag_tid=8, entry_tid=29991' 2026-02-22T19:57:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:12.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:12.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:12.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:12.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=119, tag_tid=8, entry_tid=29991' 2026-02-22T19:57:12.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:12.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29991 2026-02-22T19:57:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 29991 2026-02-22T19:57:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:13.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:13.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:13.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:13.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:13.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=119, tag_tid=8, entry_tid=30231], [object_number=118, tag_tid=8, entry_tid=30230], [object_number=117, tag_tid=8, entry_tid=30229], [object_number=116, tag_tid=8, entry_tid=30228]]], state=connected] 2026-02-22T19:57:13.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:13.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:13.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:13.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:13.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:13.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:13.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=119, tag_tid=8, entry_tid=30231], [object_number=118, tag_tid=8, entry_tid=30230], [object_number=117, tag_tid=8, entry_tid=30229], [object_number=116, tag_tid=8, entry_tid=30228]]], state=connected] 2026-02-22T19:57:13.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=119, tag_tid=8, entry_tid=30231' 2026-02-22T19:57:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=119, tag_tid=8, entry_tid=30231' 2026-02-22T19:57:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=119, tag_tid=8, entry_tid=30231' '!=' 'object_number=119, tag_tid=8, entry_tid=29991' 2026-02-22T19:57:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=119, tag_tid=8, entry_tid=30231' = 'object_number=119, tag_tid=8, entry_tid=29991' 2026-02-22T19:57:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=119, tag_tid=8, entry_tid=30231' 2026-02-22T19:57:13.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:13.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:13.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:13.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=119, tag_tid=8, entry_tid=30231' 2026-02-22T19:57:13.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:13.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:13.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=119, tag_tid=8, entry_tid=30231' 2026-02-22T19:57:13.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:13.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30231 2026-02-22T19:57:13.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 30231 2026-02-22T19:57:13.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:13.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:13.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:13.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:13.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:13.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:13.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:13.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:13.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:13.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:13.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:13.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:13.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:13.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:13.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:13.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:13.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:13.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:13.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:13.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:13.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:13.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:13.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:13.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:13.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:13.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:13.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:13.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:13.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:13.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:13.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:13.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:13.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:13.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:13.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:13.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=118, tag_tid=8, entry_tid=30446], [object_number=117, tag_tid=8, entry_tid=30445], [object_number=116, tag_tid=8, entry_tid=30444], [object_number=119, tag_tid=8, entry_tid=30443]]], state=connected] 2026-02-22T19:57:13.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:13.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:13.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:13.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:13.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:13.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:13.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=118, tag_tid=8, entry_tid=30446], [object_number=117, tag_tid=8, entry_tid=30445], [object_number=116, tag_tid=8, entry_tid=30444], [object_number=119, tag_tid=8, entry_tid=30443]]], state=connected] 2026-02-22T19:57:13.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=118, tag_tid=8, entry_tid=30446' 2026-02-22T19:57:13.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=118, tag_tid=8, entry_tid=30446' 2026-02-22T19:57:13.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=118, tag_tid=8, entry_tid=30446' '!=' 'object_number=119, tag_tid=8, entry_tid=30231' 2026-02-22T19:57:13.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:13.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=118, tag_tid=8, entry_tid=30446' = 'object_number=119, tag_tid=8, entry_tid=30231' 2026-02-22T19:57:13.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=118, tag_tid=8, entry_tid=30446' 2026-02-22T19:57:13.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:13.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:13.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:13.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=118, tag_tid=8, entry_tid=30446' 2026-02-22T19:57:13.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:13.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:13.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=118, tag_tid=8, entry_tid=30446' 2026-02-22T19:57:13.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:13.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30446 2026-02-22T19:57:13.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 30446 2026-02-22T19:57:13.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:13.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:13.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:13.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:13.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:13.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:13.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:13.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:13.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:13.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:13.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:13.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=117, tag_tid=8, entry_tid=30669], [object_number=116, tag_tid=8, entry_tid=30668], [object_number=119, tag_tid=8, entry_tid=30667], [object_number=118, tag_tid=8, entry_tid=30666]]], state=connected] 2026-02-22T19:57:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:13.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:13.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:13.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=117, tag_tid=8, entry_tid=30669], [object_number=116, tag_tid=8, entry_tid=30668], [object_number=119, tag_tid=8, entry_tid=30667], [object_number=118, tag_tid=8, entry_tid=30666]]], state=connected] 2026-02-22T19:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:13.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=117, tag_tid=8, entry_tid=30669' 2026-02-22T19:57:13.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=117, tag_tid=8, entry_tid=30669' 2026-02-22T19:57:13.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=117, tag_tid=8, entry_tid=30669' '!=' 'object_number=118, tag_tid=8, entry_tid=30446' 2026-02-22T19:57:13.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:13.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=117, tag_tid=8, entry_tid=30669' = 'object_number=118, tag_tid=8, entry_tid=30446' 2026-02-22T19:57:13.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=117, tag_tid=8, entry_tid=30669' 2026-02-22T19:57:13.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:13.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:13.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=8, entry_tid=30669' 2026-02-22T19:57:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:13.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:13.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:13.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:14.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=8, entry_tid=30669' 2026-02-22T19:57:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30669 2026-02-22T19:57:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 30669 2026-02-22T19:57:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:14.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:14.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:14.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:14.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:14.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:14.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:14.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:14.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:14.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:14.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:14.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:14.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:14.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:14.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:14.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:14.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=117, tag_tid=8, entry_tid=30893], [object_number=116, tag_tid=8, entry_tid=30892], [object_number=119, tag_tid=8, entry_tid=30891], [object_number=118, tag_tid=8, entry_tid=30890]]], state=connected] 2026-02-22T19:57:14.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:14.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:14.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:14.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:14.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:14.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:14.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:14.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:14.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:14.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:14.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:14.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=117, tag_tid=8, entry_tid=30893], [object_number=116, tag_tid=8, entry_tid=30892], [object_number=119, tag_tid=8, entry_tid=30891], [object_number=118, tag_tid=8, entry_tid=30890]]], state=connected] 2026-02-22T19:57:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=117, tag_tid=8, entry_tid=30893' 2026-02-22T19:57:14.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=117, tag_tid=8, entry_tid=30893' 2026-02-22T19:57:14.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=117, tag_tid=8, entry_tid=30893' '!=' 'object_number=117, tag_tid=8, entry_tid=30669' 2026-02-22T19:57:14.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=117, tag_tid=8, entry_tid=30893' = 'object_number=117, tag_tid=8, entry_tid=30669' 2026-02-22T19:57:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=117, tag_tid=8, entry_tid=30893' 2026-02-22T19:57:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=8, entry_tid=30893' 2026-02-22T19:57:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:14.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:14.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:14.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=8, entry_tid=30893' 2026-02-22T19:57:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30893 2026-02-22T19:57:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 30893 2026-02-22T19:57:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:14.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:14.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:14.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:14.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:14.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:14.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:14.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:14.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:14.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:14.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:14.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:14.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:14.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:14.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:14.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:14.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:14.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:14.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:14.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:14.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:14.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:14.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=118, tag_tid=8, entry_tid=31106], [object_number=117, tag_tid=8, entry_tid=31105], [object_number=116, tag_tid=8, entry_tid=31104], [object_number=119, tag_tid=8, entry_tid=31103]]], state=connected] 2026-02-22T19:57:14.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:14.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:14.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:14.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:14.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:14.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:14.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=118, tag_tid=8, entry_tid=31106], [object_number=117, tag_tid=8, entry_tid=31105], [object_number=116, tag_tid=8, entry_tid=31104], [object_number=119, tag_tid=8, entry_tid=31103]]], state=connected] 2026-02-22T19:57:14.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=118, tag_tid=8, entry_tid=31106' 2026-02-22T19:57:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=118, tag_tid=8, entry_tid=31106' 2026-02-22T19:57:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=118, tag_tid=8, entry_tid=31106' '!=' 'object_number=117, tag_tid=8, entry_tid=30893' 2026-02-22T19:57:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:57:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=118, tag_tid=8, entry_tid=31106' = 'object_number=117, tag_tid=8, entry_tid=30893' 2026-02-22T19:57:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=118, tag_tid=8, entry_tid=31106' 2026-02-22T19:57:14.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:14.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:14.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=9 2026-02-22T19:57:14.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=118, tag_tid=8, entry_tid=31106' 2026-02-22T19:57:14.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:57:14.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=8 2026-02-22T19:57:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:57:14.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=118, tag_tid=8, entry_tid=31106' 2026-02-22T19:57:14.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:57:14.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:57:14.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31106 2026-02-22T19:57:14.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 9 = 8 -a 1 -le 31106 2026-02-22T19:57:14.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:57:14.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:14.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:57:14.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:14.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:57:14.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:14.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:14.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:14.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:14.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:57:15.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:57:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:15.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:15.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:15.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:15.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:15.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:15.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:15.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:15.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:15.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:57:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:57:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:57:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:57:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:57:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:57:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:57:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:57:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 29 2026-02-22T19:57:15.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 29 2026-02-22T19:57:15.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:57:15.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:15.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=117, tag_tid=9, entry_tid=1], [object_number=116, tag_tid=9, entry_tid=0], [object_number=119, tag_tid=8, entry_tid=31139], [object_number=118, tag_tid=8, entry_tid=31138]]], state=connected] 2026-02-22T19:57:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:57:15.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=117, tag_tid=9, entry_tid=1' -a 'object_number=117, tag_tid=9, entry_tid=1' = 'object_number=117, tag_tid=9, entry_tid=1' 2026-02-22T19:57:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2026-02-22T19:57:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap3 2026-02-22T19:57:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:57:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T19:57:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:57:16.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 5 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T19:57:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T19:57:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T19:57:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 19:55:26 2026 2026-02-22T19:57:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 19:56:38 2026 2026-02-22T19:57:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T19:57:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T19:57:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T19:57:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T19:57:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:57:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T19:57:16.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n snap2-clean ']' 2026-02-22T19:57:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap2-clean 2026-02-22T19:57:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap2-clean 2026-02-22T19:57:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T19:57:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:57:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:57:16.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:57:16.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:17.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:57:17.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:17.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap3 2026-02-22T19:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap3 2026-02-22T19:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T19:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:57:17.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:57:18.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:18.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:57:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:18.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:57:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap3-clean 2026-02-22T19:57:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap3-clean 2026-02-22T19:57:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:57:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:18.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:18.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap3-clean 2026-02-22T19:57:18.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T19:57:19.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T19:57:19.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T19:57:19.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:57:19.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:19.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:19.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T19:57:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T19:57:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T19:57:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T19:57:20.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 64165888 bytes 2026-02-22T19:57:21.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 113323008 bytes 2026-02-22T19:57:21.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 136539136 bytes 2026-02-22T19:57:22.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 164025344 bytes 2026-02-22T19:57:22.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 202284032 bytes 2026-02-22T19:57:23.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 211429376 bytes 2026-02-22T19:57:23.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 254862336 bytes 2026-02-22T19:57:24.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 331877376 bytes 2026-02-22T19:57:25.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 384299008 bytes 2026-02-22T19:57:25.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 386323456 bytes 2026-02-22T19:57:25.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 417408000 bytes 2026-02-22T19:57:26.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 495879168 bytes 2026-02-22T19:57:27.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 551911424 bytes 2026-02-22T19:57:27.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 576630784 bytes 2026-02-22T19:57:28.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 595173376 bytes 2026-02-22T19:57:28.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 626331648 bytes 2026-02-22T19:57:29.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1700 IOs, 699350016 bytes 2026-02-22T19:57:30.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1800 IOs, 739338240 bytes 2026-02-22T19:57:30.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1900 IOs, 739940352 bytes 2026-02-22T19:57:30.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2000 IOs, 752086016 bytes 2026-02-22T19:57:31.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2100 IOs, 795188224 bytes 2026-02-22T19:57:31.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2200 IOs, 810250240 bytes 2026-02-22T19:57:31.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T19:57:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T19:57:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T19:57:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:57:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:57:31.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:57:31.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:31.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:57:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:57:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:57:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:57:32.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:57:32.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:57:32.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:57:32.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T19:57:32.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:57:32.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T19:57:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":10222323.47,"entries_behind_primary":7108,"entries_per_second":642.33,"non_primary_position":{"entry_tid":24032,"object_number":108,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":117,"tag_tid":9},"seconds_until_synced":11} 2026-02-22T19:57:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.0 on trial066 2026-02-22T19:57:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:57:03 2026-02-22T19:57:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T19:57:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T19:57:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T19:57:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T19:57:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:57:03' 2026-02-22T19:57:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:57:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T19:57:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T19:57:32.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":10222323.47,"entries_behind_primary":7108,"entries_per_second":642.33,"non_primary_position":{"entry_tid":24032,"object_number":108,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":117,"tag_tid":9},"seconds_until_synced":11} 2026-02-22T19:57:32.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:57:32.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:57:32.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T19:57:32.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:57:32.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:32.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.0 on trial066 2026-02-22T19:57:32.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T19:57:32.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:32.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:57:03' 2026-02-22T19:57:32.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:57:03' 2026-02-22T19:57:32.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:57:32.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:32.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:32.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:32.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 19:57:03' 2026-02-22T19:57:32.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:57:32.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:57:32.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:32.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142024257mirror.0trial06619.2.3-689-gf90f4eaetrueOK42134266mirror.3trial06619.2.3-689-gf90f4eaefalseOK43154318mirror.1trial06619.2.3-689-gf90f4eaefalseOK43504353mirror.2trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":10222323.47,"entries_behind_primary":7108,"entries_per_second":642.33,"non_primary_position":{"entry_tid":24032,"object_number":108,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":117,"tag_tid":9},"seconds_until_synced":11}42024257mirror.0trial0662026-02-22 19:57:03cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 19:57:03' 2026-02-22T19:57:32.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T19:57:32.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:32.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:57:03' 2026-02-22T19:57:32.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:32.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T19:57:32.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:57:32.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T19:57:32.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:57:32.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:32.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:57:32.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap4 2026-02-22T19:57:32.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap4 2026-02-22T19:57:32.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:57:32.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:57:32.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:57:32.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap4 2026-02-22T19:57:32.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2026-02-22T19:58:02.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:58:02.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:02.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:02.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:02.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:58:02.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:02.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:02.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:02.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T19:58:02.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:58:02.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:02.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:58:03.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:58:03.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:03.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:03.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:03.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T19:58:03.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T19:58:03.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T19:58:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:58:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T19:58:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T19:58:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:58:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T19:58:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T19:58:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T19:58:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:58:03.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:58:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:03.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:03.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T19:58:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:58:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:58:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T19:58:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T19:58:03.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T19:58:03.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T19:58:03.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:03.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:58:03.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:58:03.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:03.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:03.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:03.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:03.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:03.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:03.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:03.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:03.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:03.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:03.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:03.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:03.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:03.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:03.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=142, tag_tid=11, entry_tid=25102], [object_number=141, tag_tid=11, entry_tid=25101], [object_number=140, tag_tid=11, entry_tid=25100], [object_number=143, tag_tid=11, entry_tid=25099]]], state=connected] 2026-02-22T19:58:04.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:04.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:04.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:04.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:04.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:04.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:04.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:04.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:04.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=142, tag_tid=11, entry_tid=25102], [object_number=141, tag_tid=11, entry_tid=25101], [object_number=140, tag_tid=11, entry_tid=25100], [object_number=143, tag_tid=11, entry_tid=25099]]], state=connected] 2026-02-22T19:58:04.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=142, tag_tid=11, entry_tid=25102' 2026-02-22T19:58:04.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=142, tag_tid=11, entry_tid=25102' 2026-02-22T19:58:04.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=142, tag_tid=11, entry_tid=25102' '!=' '' 2026-02-22T19:58:04.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:04.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=142, tag_tid=11, entry_tid=25102' = '' 2026-02-22T19:58:04.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=142, tag_tid=11, entry_tid=25102' 2026-02-22T19:58:04.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:04.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:04.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=142, tag_tid=11, entry_tid=25102' 2026-02-22T19:58:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:04.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:04.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:04.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:04.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=142, tag_tid=11, entry_tid=25102' 2026-02-22T19:58:04.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:04.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25102 2026-02-22T19:58:04.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 25102 2026-02-22T19:58:04.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:04.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:04.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:04.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:04.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:04.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:04.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:04.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:04.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:04.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:04.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:04.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:04.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:04.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:04.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:04.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:04.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:04.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:04.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:04.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:04.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:04.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:04.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:04.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:04.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:04.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:04.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:04.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=140, tag_tid=11, entry_tid=25452], [object_number=143, tag_tid=11, entry_tid=25451], [object_number=142, tag_tid=11, entry_tid=25450], [object_number=141, tag_tid=11, entry_tid=25449]]], state=connected] 2026-02-22T19:58:04.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:04.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:04.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:04.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:04.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:04.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:04.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:04.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:04.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:04.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:04.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=140, tag_tid=11, entry_tid=25452], [object_number=143, tag_tid=11, entry_tid=25451], [object_number=142, tag_tid=11, entry_tid=25450], [object_number=141, tag_tid=11, entry_tid=25449]]], state=connected] 2026-02-22T19:58:04.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=140, tag_tid=11, entry_tid=25452' 2026-02-22T19:58:04.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=140, tag_tid=11, entry_tid=25452' 2026-02-22T19:58:04.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=140, tag_tid=11, entry_tid=25452' '!=' 'object_number=142, tag_tid=11, entry_tid=25102' 2026-02-22T19:58:04.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:04.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=140, tag_tid=11, entry_tid=25452' = 'object_number=142, tag_tid=11, entry_tid=25102' 2026-02-22T19:58:04.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=140, tag_tid=11, entry_tid=25452' 2026-02-22T19:58:04.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:04.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:04.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:04.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=140, tag_tid=11, entry_tid=25452' 2026-02-22T19:58:04.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:04.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:04.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:04.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:04.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=140, tag_tid=11, entry_tid=25452' 2026-02-22T19:58:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:04.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25452 2026-02-22T19:58:04.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 25452 2026-02-22T19:58:04.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:04.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:04.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:04.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:04.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:04.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:04.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:04.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:04.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:04.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:04.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:04.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:04.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:04.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:04.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=140, tag_tid=11, entry_tid=25656], [object_number=143, tag_tid=11, entry_tid=25655], [object_number=142, tag_tid=11, entry_tid=25654], [object_number=141, tag_tid=11, entry_tid=25653]]], state=connected] 2026-02-22T19:58:04.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:04.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:04.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:04.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:04.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:04.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:04.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:04.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=140, tag_tid=11, entry_tid=25656], [object_number=143, tag_tid=11, entry_tid=25655], [object_number=142, tag_tid=11, entry_tid=25654], [object_number=141, tag_tid=11, entry_tid=25653]]], state=connected] 2026-02-22T19:58:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=140, tag_tid=11, entry_tid=25656' 2026-02-22T19:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=140, tag_tid=11, entry_tid=25656' 2026-02-22T19:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=140, tag_tid=11, entry_tid=25656' '!=' 'object_number=140, tag_tid=11, entry_tid=25452' 2026-02-22T19:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=140, tag_tid=11, entry_tid=25656' = 'object_number=140, tag_tid=11, entry_tid=25452' 2026-02-22T19:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=140, tag_tid=11, entry_tid=25656' 2026-02-22T19:58:04.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:04.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:04.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:04.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=140, tag_tid=11, entry_tid=25656' 2026-02-22T19:58:04.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:04.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:04.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:04.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:04.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:04.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=140, tag_tid=11, entry_tid=25656' 2026-02-22T19:58:04.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:04.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25656 2026-02-22T19:58:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 25656 2026-02-22T19:58:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:05.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:05.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:05.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:05.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:05.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:05.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:05.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:05.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:05.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:05.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:05.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=142, tag_tid=11, entry_tid=25870], [object_number=141, tag_tid=11, entry_tid=25869], [object_number=140, tag_tid=11, entry_tid=25868], [object_number=143, tag_tid=11, entry_tid=25867]]], state=connected] 2026-02-22T19:58:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:05.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:05.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:05.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:05.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:05.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:05.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:05.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:05.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=142, tag_tid=11, entry_tid=25870], [object_number=141, tag_tid=11, entry_tid=25869], [object_number=140, tag_tid=11, entry_tid=25868], [object_number=143, tag_tid=11, entry_tid=25867]]], state=connected] 2026-02-22T19:58:05.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=142, tag_tid=11, entry_tid=25870' 2026-02-22T19:58:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=142, tag_tid=11, entry_tid=25870' 2026-02-22T19:58:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=142, tag_tid=11, entry_tid=25870' '!=' 'object_number=140, tag_tid=11, entry_tid=25656' 2026-02-22T19:58:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=142, tag_tid=11, entry_tid=25870' = 'object_number=140, tag_tid=11, entry_tid=25656' 2026-02-22T19:58:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=142, tag_tid=11, entry_tid=25870' 2026-02-22T19:58:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:05.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:05.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=142, tag_tid=11, entry_tid=25870' 2026-02-22T19:58:05.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:05.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:05.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:05.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=142, tag_tid=11, entry_tid=25870' 2026-02-22T19:58:05.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:05.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:05.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25870 2026-02-22T19:58:05.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 25870 2026-02-22T19:58:05.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:05.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:05.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:05.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:05.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:05.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:05.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:05.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:05.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:05.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:05.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:05.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:05.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:05.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:05.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:05.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:05.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:05.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:05.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=140, tag_tid=11, entry_tid=26088], [object_number=143, tag_tid=11, entry_tid=26087], [object_number=142, tag_tid=11, entry_tid=26086], [object_number=141, tag_tid=11, entry_tid=26085]]], state=connected] 2026-02-22T19:58:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:05.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:05.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:05.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:05.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:05.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=140, tag_tid=11, entry_tid=26088], [object_number=143, tag_tid=11, entry_tid=26087], [object_number=142, tag_tid=11, entry_tid=26086], [object_number=141, tag_tid=11, entry_tid=26085]]], state=connected] 2026-02-22T19:58:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=140, tag_tid=11, entry_tid=26088' 2026-02-22T19:58:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=140, tag_tid=11, entry_tid=26088' 2026-02-22T19:58:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=140, tag_tid=11, entry_tid=26088' '!=' 'object_number=142, tag_tid=11, entry_tid=25870' 2026-02-22T19:58:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=140, tag_tid=11, entry_tid=26088' = 'object_number=142, tag_tid=11, entry_tid=25870' 2026-02-22T19:58:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=140, tag_tid=11, entry_tid=26088' 2026-02-22T19:58:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:05.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:05.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=140, tag_tid=11, entry_tid=26088' 2026-02-22T19:58:05.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:05.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:05.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:05.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:05.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:05.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:05.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:05.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=140, tag_tid=11, entry_tid=26088' 2026-02-22T19:58:05.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:05.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:05.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26088 2026-02-22T19:58:05.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 26088 2026-02-22T19:58:05.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:05.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:05.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:05.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:05.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:05.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:05.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:05.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:05.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:05.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:06.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:06.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=140, tag_tid=11, entry_tid=26320], [object_number=143, tag_tid=11, entry_tid=26319], [object_number=142, tag_tid=11, entry_tid=26318], [object_number=141, tag_tid=11, entry_tid=26317]]], state=connected] 2026-02-22T19:58:06.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:06.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:06.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:06.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:06.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:06.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:06.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:06.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:06.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:06.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:06.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=140, tag_tid=11, entry_tid=26320], [object_number=143, tag_tid=11, entry_tid=26319], [object_number=142, tag_tid=11, entry_tid=26318], [object_number=141, tag_tid=11, entry_tid=26317]]], state=connected] 2026-02-22T19:58:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=140, tag_tid=11, entry_tid=26320' 2026-02-22T19:58:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=140, tag_tid=11, entry_tid=26320' 2026-02-22T19:58:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=140, tag_tid=11, entry_tid=26320' '!=' 'object_number=140, tag_tid=11, entry_tid=26088' 2026-02-22T19:58:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=140, tag_tid=11, entry_tid=26320' = 'object_number=140, tag_tid=11, entry_tid=26088' 2026-02-22T19:58:06.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=140, tag_tid=11, entry_tid=26320' 2026-02-22T19:58:06.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:06.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:06.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:06.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=140, tag_tid=11, entry_tid=26320' 2026-02-22T19:58:06.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:06.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:06.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:06.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:06.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=140, tag_tid=11, entry_tid=26320' 2026-02-22T19:58:06.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:06.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26320 2026-02-22T19:58:06.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 26320 2026-02-22T19:58:06.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:06.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:06.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:06.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:06.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:06.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:06.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:06.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:06.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:06.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:06.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:06.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:06.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:06.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:06.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:06.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:06.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:06.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:06.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:06.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:06.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:06.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:06.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:06.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:06.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:06.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=142, tag_tid=11, entry_tid=26578], [object_number=141, tag_tid=11, entry_tid=26577], [object_number=140, tag_tid=11, entry_tid=26576], [object_number=143, tag_tid=11, entry_tid=26575]]], state=connected] 2026-02-22T19:58:06.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:06.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:06.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:06.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=142, tag_tid=11, entry_tid=26578], [object_number=141, tag_tid=11, entry_tid=26577], [object_number=140, tag_tid=11, entry_tid=26576], [object_number=143, tag_tid=11, entry_tid=26575]]], state=connected] 2026-02-22T19:58:06.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=142, tag_tid=11, entry_tid=26578' 2026-02-22T19:58:06.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=142, tag_tid=11, entry_tid=26578' 2026-02-22T19:58:06.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=142, tag_tid=11, entry_tid=26578' '!=' 'object_number=140, tag_tid=11, entry_tid=26320' 2026-02-22T19:58:06.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:06.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=142, tag_tid=11, entry_tid=26578' = 'object_number=140, tag_tid=11, entry_tid=26320' 2026-02-22T19:58:06.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=142, tag_tid=11, entry_tid=26578' 2026-02-22T19:58:06.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:06.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:06.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:06.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=142, tag_tid=11, entry_tid=26578' 2026-02-22T19:58:06.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:06.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:06.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:06.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:06.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:06.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=142, tag_tid=11, entry_tid=26578' 2026-02-22T19:58:06.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:06.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26578 2026-02-22T19:58:06.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 26578 2026-02-22T19:58:06.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:06.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:06.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:06.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:06.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:06.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:06.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:06.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:06.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:06.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:06.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:06.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:06.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:06.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:06.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:06.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:06.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:06.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:06.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:06.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:06.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:06.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:06.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:06.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:06.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:06.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:06.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:06.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:06.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:06.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:06.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:06.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:06.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:06.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:06.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:06.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:06.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:06.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:06.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:06.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:06.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:06.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=141, tag_tid=11, entry_tid=26821], [object_number=140, tag_tid=11, entry_tid=26820], [object_number=143, tag_tid=11, entry_tid=26819], [object_number=142, tag_tid=11, entry_tid=26818]]], state=connected] 2026-02-22T19:58:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:06.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:06.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:06.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:06.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:06.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:06.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:06.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:06.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:06.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:06.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:06.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:06.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:06.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:06.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:06.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=141, tag_tid=11, entry_tid=26821], [object_number=140, tag_tid=11, entry_tid=26820], [object_number=143, tag_tid=11, entry_tid=26819], [object_number=142, tag_tid=11, entry_tid=26818]]], state=connected] 2026-02-22T19:58:06.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=141, tag_tid=11, entry_tid=26821' 2026-02-22T19:58:06.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=141, tag_tid=11, entry_tid=26821' 2026-02-22T19:58:06.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=141, tag_tid=11, entry_tid=26821' '!=' 'object_number=142, tag_tid=11, entry_tid=26578' 2026-02-22T19:58:06.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:06.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=141, tag_tid=11, entry_tid=26821' = 'object_number=142, tag_tid=11, entry_tid=26578' 2026-02-22T19:58:06.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=141, tag_tid=11, entry_tid=26821' 2026-02-22T19:58:06.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:06.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:06.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:06.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=141, tag_tid=11, entry_tid=26821' 2026-02-22T19:58:06.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:06.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=141, tag_tid=11, entry_tid=26821' 2026-02-22T19:58:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:06.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26821 2026-02-22T19:58:06.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 26821 2026-02-22T19:58:06.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:06.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:06.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:07.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:07.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:07.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:07.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:07.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:07.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:07.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:07.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:07.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:07.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:07.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:07.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:07.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:07.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=143, tag_tid=11, entry_tid=27039], [object_number=142, tag_tid=11, entry_tid=27038], [object_number=141, tag_tid=11, entry_tid=27037], [object_number=140, tag_tid=11, entry_tid=27036]]], state=connected] 2026-02-22T19:58:07.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:07.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:07.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:07.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:07.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:07.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:07.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:07.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:07.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:07.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:07.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:07.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:07.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=143, tag_tid=11, entry_tid=27039], [object_number=142, tag_tid=11, entry_tid=27038], [object_number=141, tag_tid=11, entry_tid=27037], [object_number=140, tag_tid=11, entry_tid=27036]]], state=connected] 2026-02-22T19:58:07.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=143, tag_tid=11, entry_tid=27039' 2026-02-22T19:58:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=143, tag_tid=11, entry_tid=27039' 2026-02-22T19:58:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=143, tag_tid=11, entry_tid=27039' '!=' 'object_number=141, tag_tid=11, entry_tid=26821' 2026-02-22T19:58:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=143, tag_tid=11, entry_tid=27039' = 'object_number=141, tag_tid=11, entry_tid=26821' 2026-02-22T19:58:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=143, tag_tid=11, entry_tid=27039' 2026-02-22T19:58:07.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:07.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:07.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:07.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=143, tag_tid=11, entry_tid=27039' 2026-02-22T19:58:07.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:07.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:07.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:07.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:07.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:07.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=143, tag_tid=11, entry_tid=27039' 2026-02-22T19:58:07.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:07.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27039 2026-02-22T19:58:07.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 27039 2026-02-22T19:58:07.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:07.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:07.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:07.360 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.756380924295219) > weight (0.25) 2026-02-22T19:58:07.360 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6761560185813551) > weight (0.25) 2026-02-22T19:58:07.360 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3022613232675394) > weight (0.25) 2026-02-22T19:58:07.360 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4992985412605765) > weight (0.25) 2026-02-22T19:58:07.360 INFO:tasks.rbd_mirror_thrash:waiting for 61.54418869132608 secs before thrashing 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:07.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:07.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:07.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:07.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:07.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:07.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:07.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:07.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:07.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=27199], [object_number=146, tag_tid=11, entry_tid=27198], [object_number=141, tag_tid=11, entry_tid=27197], [object_number=140, tag_tid=11, entry_tid=27196]]], state=connected] 2026-02-22T19:58:07.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:07.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:07.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:07.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 35 2026-02-22T19:58:07.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:07.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:07.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:07.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=27199], [object_number=146, tag_tid=11, entry_tid=27198], [object_number=141, tag_tid=11, entry_tid=27197], [object_number=140, tag_tid=11, entry_tid=27196]]], state=connected] 2026-02-22T19:58:07.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=147, tag_tid=11, entry_tid=27199' 2026-02-22T19:58:07.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=147, tag_tid=11, entry_tid=27199' 2026-02-22T19:58:07.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=27199' '!=' 'object_number=143, tag_tid=11, entry_tid=27039' 2026-02-22T19:58:07.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:07.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=27199' = 'object_number=143, tag_tid=11, entry_tid=27039' 2026-02-22T19:58:07.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=147, tag_tid=11, entry_tid=27199' 2026-02-22T19:58:07.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:07.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:07.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:07.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=27199' 2026-02-22T19:58:07.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:07.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:07.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:07.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:07.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:07.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=27199' 2026-02-22T19:58:07.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:07.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27199 2026-02-22T19:58:07.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 27199 2026-02-22T19:58:07.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:07.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:07.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:07.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:07.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:07.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:07.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:07.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:07.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:07.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:07.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:07.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:07.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:07.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:07.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:07.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:07.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:07.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:07.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:07.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:07.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:07.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:07.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:07.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:07.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:07.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:07.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:07.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:07.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:07.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:07.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:07.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:07.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:07.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=27403], [object_number=146, tag_tid=11, entry_tid=27402], [object_number=145, tag_tid=11, entry_tid=27401], [object_number=144, tag_tid=11, entry_tid=27400]]], state=connected] 2026-02-22T19:58:07.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:07.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=27403], [object_number=146, tag_tid=11, entry_tid=27402], [object_number=145, tag_tid=11, entry_tid=27401], [object_number=144, tag_tid=11, entry_tid=27400]]], state=connected] 2026-02-22T19:58:07.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=147, tag_tid=11, entry_tid=27403' 2026-02-22T19:58:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=147, tag_tid=11, entry_tid=27403' 2026-02-22T19:58:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=27403' '!=' 'object_number=147, tag_tid=11, entry_tid=27199' 2026-02-22T19:58:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=27403' = 'object_number=147, tag_tid=11, entry_tid=27199' 2026-02-22T19:58:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=147, tag_tid=11, entry_tid=27403' 2026-02-22T19:58:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:07.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=27403' 2026-02-22T19:58:07.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:07.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=27403' 2026-02-22T19:58:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:07.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27403 2026-02-22T19:58:07.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 27403 2026-02-22T19:58:07.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:07.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:07.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:08.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:08.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:08.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:08.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:08.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:08.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:08.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=27627], [object_number=146, tag_tid=11, entry_tid=27626], [object_number=145, tag_tid=11, entry_tid=27625], [object_number=144, tag_tid=11, entry_tid=27624]]], state=connected] 2026-02-22T19:58:08.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:08.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:08.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:08.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:08.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:08.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:08.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:08.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:08.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:08.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=27627], [object_number=146, tag_tid=11, entry_tid=27626], [object_number=145, tag_tid=11, entry_tid=27625], [object_number=144, tag_tid=11, entry_tid=27624]]], state=connected] 2026-02-22T19:58:08.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=147, tag_tid=11, entry_tid=27627' 2026-02-22T19:58:08.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=147, tag_tid=11, entry_tid=27627' 2026-02-22T19:58:08.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=27627' '!=' 'object_number=147, tag_tid=11, entry_tid=27403' 2026-02-22T19:58:08.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:08.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=27627' = 'object_number=147, tag_tid=11, entry_tid=27403' 2026-02-22T19:58:08.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=147, tag_tid=11, entry_tid=27627' 2026-02-22T19:58:08.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:08.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:08.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:08.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:08.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=27627' 2026-02-22T19:58:08.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:08.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:08.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:08.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:08.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:08.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:08.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:08.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=27627' 2026-02-22T19:58:08.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:08.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:08.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27627 2026-02-22T19:58:08.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 27627 2026-02-22T19:58:08.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:08.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:08.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:08.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:08.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:08.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:08.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:08.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:08.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:08.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:08.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:08.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:08.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:08.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:08.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:08.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:08.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:08.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:08.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:08.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:08.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:08.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:08.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:08.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:08.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:08.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:08.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:08.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:08.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:08.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:08.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=27971], [object_number=146, tag_tid=11, entry_tid=27970], [object_number=145, tag_tid=11, entry_tid=27969], [object_number=144, tag_tid=11, entry_tid=27968]]], state=connected] 2026-02-22T19:58:08.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:08.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:08.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:08.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:08.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=27971], [object_number=146, tag_tid=11, entry_tid=27970], [object_number=145, tag_tid=11, entry_tid=27969], [object_number=144, tag_tid=11, entry_tid=27968]]], state=connected] 2026-02-22T19:58:08.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=147, tag_tid=11, entry_tid=27971' 2026-02-22T19:58:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=147, tag_tid=11, entry_tid=27971' 2026-02-22T19:58:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=27971' '!=' 'object_number=147, tag_tid=11, entry_tid=27627' 2026-02-22T19:58:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=27971' = 'object_number=147, tag_tid=11, entry_tid=27627' 2026-02-22T19:58:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=147, tag_tid=11, entry_tid=27971' 2026-02-22T19:58:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:08.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:08.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=27971' 2026-02-22T19:58:08.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:08.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:08.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:08.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:08.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:08.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:08.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:08.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=27971' 2026-02-22T19:58:08.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:08.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:08.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27971 2026-02-22T19:58:08.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 27971 2026-02-22T19:58:08.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:08.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:08.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:09.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:09.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:09.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:09.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:09.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:09.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:09.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:09.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:09.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:09.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:09.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:09.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:09.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:09.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:09.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:09.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:09.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=28195], [object_number=146, tag_tid=11, entry_tid=28194], [object_number=145, tag_tid=11, entry_tid=28193], [object_number=144, tag_tid=11, entry_tid=28192]]], state=connected] 2026-02-22T19:58:09.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:09.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:09.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:09.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:09.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:09.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:09.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:09.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:09.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:09.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=28195], [object_number=146, tag_tid=11, entry_tid=28194], [object_number=145, tag_tid=11, entry_tid=28193], [object_number=144, tag_tid=11, entry_tid=28192]]], state=connected] 2026-02-22T19:58:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=147, tag_tid=11, entry_tid=28195' 2026-02-22T19:58:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=147, tag_tid=11, entry_tid=28195' 2026-02-22T19:58:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=28195' '!=' 'object_number=147, tag_tid=11, entry_tid=27971' 2026-02-22T19:58:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=28195' = 'object_number=147, tag_tid=11, entry_tid=27971' 2026-02-22T19:58:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=147, tag_tid=11, entry_tid=28195' 2026-02-22T19:58:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:09.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=28195' 2026-02-22T19:58:09.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:09.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:09.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:09.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:09.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:09.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=28195' 2026-02-22T19:58:09.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:09.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28195 2026-02-22T19:58:09.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 28195 2026-02-22T19:58:09.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:09.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:09.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:09.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:09.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:09.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:09.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:09.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:09.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=28430], [object_number=145, tag_tid=11, entry_tid=28429], [object_number=144, tag_tid=11, entry_tid=28428], [object_number=147, tag_tid=11, entry_tid=28427]]], state=connected] 2026-02-22T19:58:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:09.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:09.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:09.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:09.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:09.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:09.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:09.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:09.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=28430], [object_number=145, tag_tid=11, entry_tid=28429], [object_number=144, tag_tid=11, entry_tid=28428], [object_number=147, tag_tid=11, entry_tid=28427]]], state=connected] 2026-02-22T19:58:09.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=146, tag_tid=11, entry_tid=28430' 2026-02-22T19:58:09.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=146, tag_tid=11, entry_tid=28430' 2026-02-22T19:58:09.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=28430' '!=' 'object_number=147, tag_tid=11, entry_tid=28195' 2026-02-22T19:58:09.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:09.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=28430' = 'object_number=147, tag_tid=11, entry_tid=28195' 2026-02-22T19:58:09.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=146, tag_tid=11, entry_tid=28430' 2026-02-22T19:58:09.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:09.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:09.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=28430' 2026-02-22T19:58:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=28430' 2026-02-22T19:58:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28430 2026-02-22T19:58:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 28430 2026-02-22T19:58:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:09.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:09.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:09.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:09.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:09.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:09.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:09.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:09.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:09.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:09.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:09.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:09.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:09.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:09.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:09.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:09.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:09.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:09.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:09.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:09.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:09.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:09.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:09.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:09.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=28634], [object_number=145, tag_tid=11, entry_tid=28633], [object_number=144, tag_tid=11, entry_tid=28632], [object_number=147, tag_tid=11, entry_tid=28631]]], state=connected] 2026-02-22T19:58:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:09.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:09.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:09.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:09.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:09.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:09.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:09.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:09.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:09.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=28634], [object_number=145, tag_tid=11, entry_tid=28633], [object_number=144, tag_tid=11, entry_tid=28632], [object_number=147, tag_tid=11, entry_tid=28631]]], state=connected] 2026-02-22T19:58:09.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:09.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=146, tag_tid=11, entry_tid=28634' 2026-02-22T19:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=146, tag_tid=11, entry_tid=28634' 2026-02-22T19:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=28634' '!=' 'object_number=146, tag_tid=11, entry_tid=28430' 2026-02-22T19:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=28634' = 'object_number=146, tag_tid=11, entry_tid=28430' 2026-02-22T19:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=146, tag_tid=11, entry_tid=28634' 2026-02-22T19:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:09.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:09.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=28634' 2026-02-22T19:58:09.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:09.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:09.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:09.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:09.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:09.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=28634' 2026-02-22T19:58:09.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:09.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:09.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28634 2026-02-22T19:58:09.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 28634 2026-02-22T19:58:09.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:09.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:09.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:10.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:10.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:10.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:10.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:10.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:10.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:10.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:10.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:10.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:10.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:10.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:10.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:10.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:10.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:10.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:10.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:10.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:10.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:10.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:10.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=145, tag_tid=11, entry_tid=28873], [object_number=144, tag_tid=11, entry_tid=28872], [object_number=147, tag_tid=11, entry_tid=28871], [object_number=146, tag_tid=11, entry_tid=28870]]], state=connected] 2026-02-22T19:58:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:10.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:10.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:10.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:10.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:10.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:10.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=145, tag_tid=11, entry_tid=28873], [object_number=144, tag_tid=11, entry_tid=28872], [object_number=147, tag_tid=11, entry_tid=28871], [object_number=146, tag_tid=11, entry_tid=28870]]], state=connected] 2026-02-22T19:58:10.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=145, tag_tid=11, entry_tid=28873' 2026-02-22T19:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=145, tag_tid=11, entry_tid=28873' 2026-02-22T19:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=145, tag_tid=11, entry_tid=28873' '!=' 'object_number=146, tag_tid=11, entry_tid=28634' 2026-02-22T19:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=145, tag_tid=11, entry_tid=28873' = 'object_number=146, tag_tid=11, entry_tid=28634' 2026-02-22T19:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=145, tag_tid=11, entry_tid=28873' 2026-02-22T19:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:10.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:10.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=145, tag_tid=11, entry_tid=28873' 2026-02-22T19:58:10.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:10.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=145, tag_tid=11, entry_tid=28873' 2026-02-22T19:58:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:10.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28873 2026-02-22T19:58:10.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 28873 2026-02-22T19:58:10.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:10.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:10.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:10.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:10.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:10.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:10.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:10.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:10.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:10.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=145, tag_tid=11, entry_tid=29081], [object_number=144, tag_tid=11, entry_tid=29080], [object_number=147, tag_tid=11, entry_tid=29079], [object_number=146, tag_tid=11, entry_tid=29078]]], state=connected] 2026-02-22T19:58:10.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:10.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:10.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:10.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:10.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:10.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:10.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=145, tag_tid=11, entry_tid=29081], [object_number=144, tag_tid=11, entry_tid=29080], [object_number=147, tag_tid=11, entry_tid=29079], [object_number=146, tag_tid=11, entry_tid=29078]]], state=connected] 2026-02-22T19:58:10.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=145, tag_tid=11, entry_tid=29081' 2026-02-22T19:58:10.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=145, tag_tid=11, entry_tid=29081' 2026-02-22T19:58:10.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=145, tag_tid=11, entry_tid=29081' '!=' 'object_number=145, tag_tid=11, entry_tid=28873' 2026-02-22T19:58:10.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:10.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=145, tag_tid=11, entry_tid=29081' = 'object_number=145, tag_tid=11, entry_tid=28873' 2026-02-22T19:58:10.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=145, tag_tid=11, entry_tid=29081' 2026-02-22T19:58:10.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:10.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:10.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:10.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:10.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=145, tag_tid=11, entry_tid=29081' 2026-02-22T19:58:10.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:10.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:10.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:10.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:10.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:10.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:10.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:10.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=145, tag_tid=11, entry_tid=29081' 2026-02-22T19:58:10.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:10.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:10.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29081 2026-02-22T19:58:10.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 29081 2026-02-22T19:58:10.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:10.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:10.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:10.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:10.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:10.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:10.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:10.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:10.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:10.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:10.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:10.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:10.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:10.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:10.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:10.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:10.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:10.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:10.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:10.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:10.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:10.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:10.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:10.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:10.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:10.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:10.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:10.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:10.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:10.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:10.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:10.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:10.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:10.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:10.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:10.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:10.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=29294], [object_number=145, tag_tid=11, entry_tid=29293], [object_number=144, tag_tid=11, entry_tid=29292], [object_number=147, tag_tid=11, entry_tid=29291]]], state=connected] 2026-02-22T19:58:11.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:11.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:11.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:11.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:11.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:11.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:11.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:11.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:11.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:11.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:11.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:11.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=29294], [object_number=145, tag_tid=11, entry_tid=29293], [object_number=144, tag_tid=11, entry_tid=29292], [object_number=147, tag_tid=11, entry_tid=29291]]], state=connected] 2026-02-22T19:58:11.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=146, tag_tid=11, entry_tid=29294' 2026-02-22T19:58:11.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=146, tag_tid=11, entry_tid=29294' 2026-02-22T19:58:11.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=29294' '!=' 'object_number=145, tag_tid=11, entry_tid=29081' 2026-02-22T19:58:11.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:11.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=29294' = 'object_number=145, tag_tid=11, entry_tid=29081' 2026-02-22T19:58:11.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=146, tag_tid=11, entry_tid=29294' 2026-02-22T19:58:11.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:11.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:11.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:11.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=29294' 2026-02-22T19:58:11.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:11.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:11.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:11.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:11.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:11.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=29294' 2026-02-22T19:58:11.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:11.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29294 2026-02-22T19:58:11.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 29294 2026-02-22T19:58:11.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:11.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:11.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:11.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:11.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:11.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:11.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:11.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:11.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:11.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:11.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:11.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:11.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:11.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:11.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:11.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:11.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:11.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:11.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:11.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:11.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:11.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:11.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:11.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:11.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:11.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:11.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:11.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=29506], [object_number=145, tag_tid=11, entry_tid=29505], [object_number=144, tag_tid=11, entry_tid=29504], [object_number=147, tag_tid=11, entry_tid=29503]]], state=connected] 2026-02-22T19:58:11.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:11.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:11.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:11.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:11.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:11.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:11.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:11.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:11.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=29506], [object_number=145, tag_tid=11, entry_tid=29505], [object_number=144, tag_tid=11, entry_tid=29504], [object_number=147, tag_tid=11, entry_tid=29503]]], state=connected] 2026-02-22T19:58:11.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=146, tag_tid=11, entry_tid=29506' 2026-02-22T19:58:11.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=146, tag_tid=11, entry_tid=29506' 2026-02-22T19:58:11.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=29506' '!=' 'object_number=146, tag_tid=11, entry_tid=29294' 2026-02-22T19:58:11.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:11.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=29506' = 'object_number=146, tag_tid=11, entry_tid=29294' 2026-02-22T19:58:11.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=146, tag_tid=11, entry_tid=29506' 2026-02-22T19:58:11.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:11.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:11.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:11.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=29506' 2026-02-22T19:58:11.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:11.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:11.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:11.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:11.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:11.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=29506' 2026-02-22T19:58:11.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:11.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29506 2026-02-22T19:58:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 29506 2026-02-22T19:58:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:11.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:11.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:11.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:11.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:11.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:11.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:11.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:11.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=29714], [object_number=145, tag_tid=11, entry_tid=29713], [object_number=144, tag_tid=11, entry_tid=29712], [object_number=147, tag_tid=11, entry_tid=29711]]], state=connected] 2026-02-22T19:58:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:11.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:11.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:11.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:11.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=29714], [object_number=145, tag_tid=11, entry_tid=29713], [object_number=144, tag_tid=11, entry_tid=29712], [object_number=147, tag_tid=11, entry_tid=29711]]], state=connected] 2026-02-22T19:58:11.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:11.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=146, tag_tid=11, entry_tid=29714' 2026-02-22T19:58:11.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=146, tag_tid=11, entry_tid=29714' 2026-02-22T19:58:11.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=29714' '!=' 'object_number=146, tag_tid=11, entry_tid=29506' 2026-02-22T19:58:11.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=29714' = 'object_number=146, tag_tid=11, entry_tid=29506' 2026-02-22T19:58:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=146, tag_tid=11, entry_tid=29714' 2026-02-22T19:58:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:11.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=29714' 2026-02-22T19:58:11.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:11.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:11.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=29714' 2026-02-22T19:58:11.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:11.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29714 2026-02-22T19:58:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 29714 2026-02-22T19:58:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:11.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:11.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:11.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:11.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:11.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:11.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:11.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:11.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:11.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:11.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:11.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:11.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:11.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:11.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:11.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:11.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:12.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:12.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:12.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:12.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:12.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:12.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:12.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:12.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:12.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:12.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:12.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:12.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:12.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=144, tag_tid=11, entry_tid=29920], [object_number=147, tag_tid=11, entry_tid=29919], [object_number=146, tag_tid=11, entry_tid=29918], [object_number=145, tag_tid=11, entry_tid=29917]]], state=connected] 2026-02-22T19:58:12.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:12.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:12.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:12.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:12.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:12.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:12.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:12.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:12.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:12.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:12.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:12.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:12.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=144, tag_tid=11, entry_tid=29920], [object_number=147, tag_tid=11, entry_tid=29919], [object_number=146, tag_tid=11, entry_tid=29918], [object_number=145, tag_tid=11, entry_tid=29917]]], state=connected] 2026-02-22T19:58:12.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=144, tag_tid=11, entry_tid=29920' 2026-02-22T19:58:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=144, tag_tid=11, entry_tid=29920' 2026-02-22T19:58:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=144, tag_tid=11, entry_tid=29920' '!=' 'object_number=146, tag_tid=11, entry_tid=29714' 2026-02-22T19:58:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=144, tag_tid=11, entry_tid=29920' = 'object_number=146, tag_tid=11, entry_tid=29714' 2026-02-22T19:58:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=144, tag_tid=11, entry_tid=29920' 2026-02-22T19:58:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:12.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:12.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:12.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=144, tag_tid=11, entry_tid=29920' 2026-02-22T19:58:12.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:12.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=144, tag_tid=11, entry_tid=29920' 2026-02-22T19:58:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29920 2026-02-22T19:58:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 29920 2026-02-22T19:58:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:12.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:12.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:12.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:12.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:12.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:12.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:12.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:12.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:12.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:12.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:12.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:12.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:12.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:12.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:12.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:12.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:12.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:12.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:12.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:12.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:12.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:12.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:12.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:12.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:12.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:12.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:12.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:12.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=145, tag_tid=11, entry_tid=30153], [object_number=144, tag_tid=11, entry_tid=30152], [object_number=147, tag_tid=11, entry_tid=30151], [object_number=146, tag_tid=11, entry_tid=30150]]], state=connected] 2026-02-22T19:58:12.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:12.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:12.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:12.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:12.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:12.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=145, tag_tid=11, entry_tid=30153], [object_number=144, tag_tid=11, entry_tid=30152], [object_number=147, tag_tid=11, entry_tid=30151], [object_number=146, tag_tid=11, entry_tid=30150]]], state=connected] 2026-02-22T19:58:12.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=145, tag_tid=11, entry_tid=30153' 2026-02-22T19:58:12.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=145, tag_tid=11, entry_tid=30153' 2026-02-22T19:58:12.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=145, tag_tid=11, entry_tid=30153' '!=' 'object_number=144, tag_tid=11, entry_tid=29920' 2026-02-22T19:58:12.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:12.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=145, tag_tid=11, entry_tid=30153' = 'object_number=144, tag_tid=11, entry_tid=29920' 2026-02-22T19:58:12.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=145, tag_tid=11, entry_tid=30153' 2026-02-22T19:58:12.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:12.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:12.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:12.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=145, tag_tid=11, entry_tid=30153' 2026-02-22T19:58:12.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:12.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:12.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:12.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:12.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:12.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=145, tag_tid=11, entry_tid=30153' 2026-02-22T19:58:12.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:12.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30153 2026-02-22T19:58:12.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 30153 2026-02-22T19:58:12.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:12.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:12.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:12.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:12.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:12.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:12.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:12.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:12.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:12.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:12.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:12.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:12.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:12.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:12.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:12.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:12.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:12.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:12.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:12.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:12.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=144, tag_tid=11, entry_tid=30384], [object_number=147, tag_tid=11, entry_tid=30383], [object_number=146, tag_tid=11, entry_tid=30382], [object_number=145, tag_tid=11, entry_tid=30381]]], state=connected] 2026-02-22T19:58:12.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:12.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=144, tag_tid=11, entry_tid=30384], [object_number=147, tag_tid=11, entry_tid=30383], [object_number=146, tag_tid=11, entry_tid=30382], [object_number=145, tag_tid=11, entry_tid=30381]]], state=connected] 2026-02-22T19:58:12.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=144, tag_tid=11, entry_tid=30384' 2026-02-22T19:58:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=144, tag_tid=11, entry_tid=30384' 2026-02-22T19:58:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=144, tag_tid=11, entry_tid=30384' '!=' 'object_number=145, tag_tid=11, entry_tid=30153' 2026-02-22T19:58:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=144, tag_tid=11, entry_tid=30384' = 'object_number=145, tag_tid=11, entry_tid=30153' 2026-02-22T19:58:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=144, tag_tid=11, entry_tid=30384' 2026-02-22T19:58:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:12.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=144, tag_tid=11, entry_tid=30384' 2026-02-22T19:58:12.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:12.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:12.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:12.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:12.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:12.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=144, tag_tid=11, entry_tid=30384' 2026-02-22T19:58:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30384 2026-02-22T19:58:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 30384 2026-02-22T19:58:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:13.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:13.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:13.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:13.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:13.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:13.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:13.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:13.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:13.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:13.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:13.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:13.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:13.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:13.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:13.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:13.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=145, tag_tid=11, entry_tid=30589], [object_number=144, tag_tid=11, entry_tid=30588], [object_number=147, tag_tid=11, entry_tid=30587], [object_number=146, tag_tid=11, entry_tid=30586]]], state=connected] 2026-02-22T19:58:13.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:13.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:13.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:13.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:13.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:13.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:13.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:13.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:13.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=145, tag_tid=11, entry_tid=30589], [object_number=144, tag_tid=11, entry_tid=30588], [object_number=147, tag_tid=11, entry_tid=30587], [object_number=146, tag_tid=11, entry_tid=30586]]], state=connected] 2026-02-22T19:58:13.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=145, tag_tid=11, entry_tid=30589' 2026-02-22T19:58:13.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=145, tag_tid=11, entry_tid=30589' 2026-02-22T19:58:13.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=145, tag_tid=11, entry_tid=30589' '!=' 'object_number=144, tag_tid=11, entry_tid=30384' 2026-02-22T19:58:13.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:13.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=145, tag_tid=11, entry_tid=30589' = 'object_number=144, tag_tid=11, entry_tid=30384' 2026-02-22T19:58:13.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=145, tag_tid=11, entry_tid=30589' 2026-02-22T19:58:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:13.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=145, tag_tid=11, entry_tid=30589' 2026-02-22T19:58:13.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:13.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:13.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=145, tag_tid=11, entry_tid=30589' 2026-02-22T19:58:13.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30589 2026-02-22T19:58:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 30589 2026-02-22T19:58:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:13.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:13.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:13.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:13.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:13.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:13.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:13.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:13.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:13.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:13.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:13.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:13.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:13.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:13.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:13.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:13.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=30791], [object_number=146, tag_tid=11, entry_tid=30790], [object_number=145, tag_tid=11, entry_tid=30789], [object_number=144, tag_tid=11, entry_tid=30788]]], state=connected] 2026-02-22T19:58:13.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:13.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:13.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:13.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:13.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:13.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:13.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:13.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:13.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:13.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=147, tag_tid=11, entry_tid=30791], [object_number=146, tag_tid=11, entry_tid=30790], [object_number=145, tag_tid=11, entry_tid=30789], [object_number=144, tag_tid=11, entry_tid=30788]]], state=connected] 2026-02-22T19:58:13.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=147, tag_tid=11, entry_tid=30791' 2026-02-22T19:58:13.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=147, tag_tid=11, entry_tid=30791' 2026-02-22T19:58:13.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=30791' '!=' 'object_number=145, tag_tid=11, entry_tid=30589' 2026-02-22T19:58:13.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:13.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=147, tag_tid=11, entry_tid=30791' = 'object_number=145, tag_tid=11, entry_tid=30589' 2026-02-22T19:58:13.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=147, tag_tid=11, entry_tid=30791' 2026-02-22T19:58:13.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:13.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:13.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=30791' 2026-02-22T19:58:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:13.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:13.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:13.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:13.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=147, tag_tid=11, entry_tid=30791' 2026-02-22T19:58:13.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:13.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30791 2026-02-22T19:58:13.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 30791 2026-02-22T19:58:13.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:13.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:13.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:13.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:13.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:13.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:13.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:13.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:13.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:13.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:13.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:13.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:13.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:13.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:13.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=30998], [object_number=145, tag_tid=11, entry_tid=30997], [object_number=144, tag_tid=11, entry_tid=30996], [object_number=147, tag_tid=11, entry_tid=30995]]], state=connected] 2026-02-22T19:58:13.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:13.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:13.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:13.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:13.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=30998], [object_number=145, tag_tid=11, entry_tid=30997], [object_number=144, tag_tid=11, entry_tid=30996], [object_number=147, tag_tid=11, entry_tid=30995]]], state=connected] 2026-02-22T19:58:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:13.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=146, tag_tid=11, entry_tid=30998' 2026-02-22T19:58:13.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=146, tag_tid=11, entry_tid=30998' 2026-02-22T19:58:13.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=30998' '!=' 'object_number=147, tag_tid=11, entry_tid=30791' 2026-02-22T19:58:13.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:13.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=30998' = 'object_number=147, tag_tid=11, entry_tid=30791' 2026-02-22T19:58:13.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=146, tag_tid=11, entry_tid=30998' 2026-02-22T19:58:13.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:13.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:13.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:13.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=30998' 2026-02-22T19:58:13.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:13.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:13.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:13.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:13.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:13.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=30998' 2026-02-22T19:58:13.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:13.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:13.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30998 2026-02-22T19:58:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 30998 2026-02-22T19:58:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:14.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:14.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:14.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:14.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:14.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:14.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:14.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:14.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:14.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:14.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:14.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:14.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:14.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=31206], [object_number=145, tag_tid=11, entry_tid=31205], [object_number=144, tag_tid=11, entry_tid=31204], [object_number=147, tag_tid=11, entry_tid=31203]]], state=connected] 2026-02-22T19:58:14.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:14.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:14.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:14.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:14.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:14.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 36 2026-02-22T19:58:14.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:14.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:14.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:14.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=146, tag_tid=11, entry_tid=31206], [object_number=145, tag_tid=11, entry_tid=31205], [object_number=144, tag_tid=11, entry_tid=31204], [object_number=147, tag_tid=11, entry_tid=31203]]], state=connected] 2026-02-22T19:58:14.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=146, tag_tid=11, entry_tid=31206' 2026-02-22T19:58:14.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=146, tag_tid=11, entry_tid=31206' 2026-02-22T19:58:14.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=31206' '!=' 'object_number=146, tag_tid=11, entry_tid=30998' 2026-02-22T19:58:14.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:14.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=146, tag_tid=11, entry_tid=31206' = 'object_number=146, tag_tid=11, entry_tid=30998' 2026-02-22T19:58:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=146, tag_tid=11, entry_tid=31206' 2026-02-22T19:58:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:14.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:14.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:14.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=31206' 2026-02-22T19:58:14.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:14.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:14.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:14.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:14.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:14.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:14.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=146, tag_tid=11, entry_tid=31206' 2026-02-22T19:58:14.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:14.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:14.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31206 2026-02-22T19:58:14.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 31206 2026-02-22T19:58:14.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:14.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:14.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:14.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:14.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:14.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:14.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:14.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:14.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:14.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:14.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:14.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:14.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:14.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:14.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:14.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:14.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:14.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=31339], [object_number=150, tag_tid=11, entry_tid=31338], [object_number=149, tag_tid=11, entry_tid=31337], [object_number=148, tag_tid=11, entry_tid=31336]]], state=connected] 2026-02-22T19:58:14.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:14.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:14.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:14.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:14.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:14.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:14.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:14.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:14.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:14.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:14.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:14.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:14.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=31339], [object_number=150, tag_tid=11, entry_tid=31338], [object_number=149, tag_tid=11, entry_tid=31337], [object_number=148, tag_tid=11, entry_tid=31336]]], state=connected] 2026-02-22T19:58:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=151, tag_tid=11, entry_tid=31339' 2026-02-22T19:58:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=151, tag_tid=11, entry_tid=31339' 2026-02-22T19:58:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=31339' '!=' 'object_number=146, tag_tid=11, entry_tid=31206' 2026-02-22T19:58:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=31339' = 'object_number=146, tag_tid=11, entry_tid=31206' 2026-02-22T19:58:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=151, tag_tid=11, entry_tid=31339' 2026-02-22T19:58:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:14.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:14.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:14.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:14.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=31339' 2026-02-22T19:58:14.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:14.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:14.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:14.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:14.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:14.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:14.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=31339' 2026-02-22T19:58:14.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:14.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31339 2026-02-22T19:58:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 31339 2026-02-22T19:58:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:14.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:14.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:14.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:14.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:14.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:14.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:14.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:14.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:14.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:14.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:14.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:14.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:14.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:14.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:14.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:14.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:14.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:14.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=31554], [object_number=149, tag_tid=11, entry_tid=31553], [object_number=148, tag_tid=11, entry_tid=31552], [object_number=151, tag_tid=11, entry_tid=31551]]], state=connected] 2026-02-22T19:58:15.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:15.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:15.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:15.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:15.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:15.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:15.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:15.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=31554], [object_number=149, tag_tid=11, entry_tid=31553], [object_number=148, tag_tid=11, entry_tid=31552], [object_number=151, tag_tid=11, entry_tid=31551]]], state=connected] 2026-02-22T19:58:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=150, tag_tid=11, entry_tid=31554' 2026-02-22T19:58:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=150, tag_tid=11, entry_tid=31554' 2026-02-22T19:58:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=31554' '!=' 'object_number=151, tag_tid=11, entry_tid=31339' 2026-02-22T19:58:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=31554' = 'object_number=151, tag_tid=11, entry_tid=31339' 2026-02-22T19:58:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=150, tag_tid=11, entry_tid=31554' 2026-02-22T19:58:15.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:15.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:15.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:15.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=31554' 2026-02-22T19:58:15.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:15.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:15.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:15.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:15.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:15.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=31554' 2026-02-22T19:58:15.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:15.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31554 2026-02-22T19:58:15.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 31554 2026-02-22T19:58:15.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:15.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:15.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:15.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:15.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:15.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:15.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:15.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:15.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:15.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:15.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:15.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:15.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:15.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:15.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:15.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:15.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:15.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:15.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:15.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=31762], [object_number=149, tag_tid=11, entry_tid=31761], [object_number=148, tag_tid=11, entry_tid=31760], [object_number=151, tag_tid=11, entry_tid=31759]]], state=connected] 2026-02-22T19:58:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:15.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:15.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:15.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:15.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:15.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:15.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:15.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:15.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:15.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:15.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:15.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:15.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=31762], [object_number=149, tag_tid=11, entry_tid=31761], [object_number=148, tag_tid=11, entry_tid=31760], [object_number=151, tag_tid=11, entry_tid=31759]]], state=connected] 2026-02-22T19:58:15.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=150, tag_tid=11, entry_tid=31762' 2026-02-22T19:58:15.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=150, tag_tid=11, entry_tid=31762' 2026-02-22T19:58:15.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=31762' '!=' 'object_number=150, tag_tid=11, entry_tid=31554' 2026-02-22T19:58:15.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:15.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=31762' = 'object_number=150, tag_tid=11, entry_tid=31554' 2026-02-22T19:58:15.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=150, tag_tid=11, entry_tid=31762' 2026-02-22T19:58:15.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:15.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:15.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:15.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=31762' 2026-02-22T19:58:15.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:15.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:15.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:15.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:15.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:15.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=31762' 2026-02-22T19:58:15.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:15.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31762 2026-02-22T19:58:15.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 31762 2026-02-22T19:58:15.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:15.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:15.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:15.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:15.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:15.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:15.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:15.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:15.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:15.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:15.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=31974], [object_number=149, tag_tid=11, entry_tid=31973], [object_number=148, tag_tid=11, entry_tid=31972], [object_number=151, tag_tid=11, entry_tid=31971]]], state=connected] 2026-02-22T19:58:15.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:15.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=31974], [object_number=149, tag_tid=11, entry_tid=31973], [object_number=148, tag_tid=11, entry_tid=31972], [object_number=151, tag_tid=11, entry_tid=31971]]], state=connected] 2026-02-22T19:58:15.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=150, tag_tid=11, entry_tid=31974' 2026-02-22T19:58:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=150, tag_tid=11, entry_tid=31974' 2026-02-22T19:58:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=31974' '!=' 'object_number=150, tag_tid=11, entry_tid=31762' 2026-02-22T19:58:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=31974' = 'object_number=150, tag_tid=11, entry_tid=31762' 2026-02-22T19:58:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=150, tag_tid=11, entry_tid=31974' 2026-02-22T19:58:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:15.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=31974' 2026-02-22T19:58:15.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:15.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:15.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:15.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:15.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:15.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=31974' 2026-02-22T19:58:15.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:15.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:15.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31974 2026-02-22T19:58:15.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 31974 2026-02-22T19:58:15.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:15.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:15.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:15.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:15.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:15.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:15.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:15.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:15.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:15.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:15.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:15.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:15.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:15.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:15.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:15.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:15.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:15.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:15.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:15.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:15.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:16.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:16.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:16.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:16.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:16.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:16.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:16.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:16.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:16.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:16.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=32245], [object_number=148, tag_tid=11, entry_tid=32244], [object_number=151, tag_tid=11, entry_tid=32243], [object_number=150, tag_tid=11, entry_tid=32242]]], state=connected] 2026-02-22T19:58:16.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:16.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:16.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:16.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:16.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:16.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:16.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:16.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:16.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=32245], [object_number=148, tag_tid=11, entry_tid=32244], [object_number=151, tag_tid=11, entry_tid=32243], [object_number=150, tag_tid=11, entry_tid=32242]]], state=connected] 2026-02-22T19:58:16.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=149, tag_tid=11, entry_tid=32245' 2026-02-22T19:58:16.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=149, tag_tid=11, entry_tid=32245' 2026-02-22T19:58:16.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=32245' '!=' 'object_number=150, tag_tid=11, entry_tid=31974' 2026-02-22T19:58:16.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:16.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=32245' = 'object_number=150, tag_tid=11, entry_tid=31974' 2026-02-22T19:58:16.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=149, tag_tid=11, entry_tid=32245' 2026-02-22T19:58:16.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:16.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:16.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:16.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=32245' 2026-02-22T19:58:16.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:16.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:16.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:16.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:16.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:16.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=32245' 2026-02-22T19:58:16.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:16.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=32245 2026-02-22T19:58:16.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 32245 2026-02-22T19:58:16.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:16.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:16.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:16.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:16.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:16.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:16.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:16.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:16.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:16.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:16.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:16.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:16.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:16.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:16.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:16.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:16.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=32463], [object_number=150, tag_tid=11, entry_tid=32462], [object_number=149, tag_tid=11, entry_tid=32461], [object_number=148, tag_tid=11, entry_tid=32460]]], state=connected] 2026-02-22T19:58:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:16.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:16.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:16.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:16.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:16.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:16.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:16.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:16.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:16.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:16.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:16.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:16.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=32463], [object_number=150, tag_tid=11, entry_tid=32462], [object_number=149, tag_tid=11, entry_tid=32461], [object_number=148, tag_tid=11, entry_tid=32460]]], state=connected] 2026-02-22T19:58:16.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=151, tag_tid=11, entry_tid=32463' 2026-02-22T19:58:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=151, tag_tid=11, entry_tid=32463' 2026-02-22T19:58:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=32463' '!=' 'object_number=149, tag_tid=11, entry_tid=32245' 2026-02-22T19:58:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=32463' = 'object_number=149, tag_tid=11, entry_tid=32245' 2026-02-22T19:58:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=151, tag_tid=11, entry_tid=32463' 2026-02-22T19:58:16.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:16.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:16.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:16.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=32463' 2026-02-22T19:58:16.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:16.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:16.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:16.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:16.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:16.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=32463' 2026-02-22T19:58:16.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:16.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=32463 2026-02-22T19:58:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 32463 2026-02-22T19:58:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:16.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:16.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:16.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:16.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:16.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:16.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:16.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:16.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:16.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:16.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:16.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:16.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:16.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:16.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:16.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:16.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:16.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:16.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:16.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:16.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=148, tag_tid=11, entry_tid=32676], [object_number=151, tag_tid=11, entry_tid=32675], [object_number=150, tag_tid=11, entry_tid=32674], [object_number=149, tag_tid=11, entry_tid=32673]]], state=connected] 2026-02-22T19:58:16.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:16.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:16.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=148, tag_tid=11, entry_tid=32676], [object_number=151, tag_tid=11, entry_tid=32675], [object_number=150, tag_tid=11, entry_tid=32674], [object_number=149, tag_tid=11, entry_tid=32673]]], state=connected] 2026-02-22T19:58:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=148, tag_tid=11, entry_tid=32676' 2026-02-22T19:58:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=148, tag_tid=11, entry_tid=32676' 2026-02-22T19:58:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=148, tag_tid=11, entry_tid=32676' '!=' 'object_number=151, tag_tid=11, entry_tid=32463' 2026-02-22T19:58:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=148, tag_tid=11, entry_tid=32676' = 'object_number=151, tag_tid=11, entry_tid=32463' 2026-02-22T19:58:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=148, tag_tid=11, entry_tid=32676' 2026-02-22T19:58:16.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:16.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:16.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:16.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=148, tag_tid=11, entry_tid=32676' 2026-02-22T19:58:16.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:16.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:16.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:16.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:16.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=148, tag_tid=11, entry_tid=32676' 2026-02-22T19:58:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=32676 2026-02-22T19:58:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 32676 2026-02-22T19:58:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:17.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:17.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:17.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:17.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:17.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:17.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:17.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:17.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:17.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:17.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:17.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:17.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:17.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:17.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:17.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:17.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:17.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:17.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:17.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=32894], [object_number=149, tag_tid=11, entry_tid=32893], [object_number=148, tag_tid=11, entry_tid=32892], [object_number=151, tag_tid=11, entry_tid=32891]]], state=connected] 2026-02-22T19:58:17.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:17.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:17.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:17.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:17.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:17.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:17.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:17.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:17.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:17.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=32894], [object_number=149, tag_tid=11, entry_tid=32893], [object_number=148, tag_tid=11, entry_tid=32892], [object_number=151, tag_tid=11, entry_tid=32891]]], state=connected] 2026-02-22T19:58:17.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=150, tag_tid=11, entry_tid=32894' 2026-02-22T19:58:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=150, tag_tid=11, entry_tid=32894' 2026-02-22T19:58:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=32894' '!=' 'object_number=148, tag_tid=11, entry_tid=32676' 2026-02-22T19:58:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=32894' = 'object_number=148, tag_tid=11, entry_tid=32676' 2026-02-22T19:58:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=150, tag_tid=11, entry_tid=32894' 2026-02-22T19:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=32894' 2026-02-22T19:58:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:17.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:17.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:17.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:17.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=32894' 2026-02-22T19:58:17.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:17.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=32894 2026-02-22T19:58:17.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 32894 2026-02-22T19:58:17.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:17.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:17.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:17.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:17.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:17.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:17.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:17.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:17.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:17.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:17.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:17.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:17.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:17.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:17.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:17.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:17.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:17.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:17.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:17.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:17.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:17.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:17.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:17.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:17.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:17.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:17.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:17.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:17.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:17.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:17.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:17.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:17.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:17.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:17.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=33101], [object_number=148, tag_tid=11, entry_tid=33100], [object_number=151, tag_tid=11, entry_tid=33099], [object_number=150, tag_tid=11, entry_tid=33098]]], state=connected] 2026-02-22T19:58:17.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:17.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:17.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:17.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=33101], [object_number=148, tag_tid=11, entry_tid=33100], [object_number=151, tag_tid=11, entry_tid=33099], [object_number=150, tag_tid=11, entry_tid=33098]]], state=connected] 2026-02-22T19:58:17.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=149, tag_tid=11, entry_tid=33101' 2026-02-22T19:58:17.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=149, tag_tid=11, entry_tid=33101' 2026-02-22T19:58:17.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=33101' '!=' 'object_number=150, tag_tid=11, entry_tid=32894' 2026-02-22T19:58:17.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:17.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=33101' = 'object_number=150, tag_tid=11, entry_tid=32894' 2026-02-22T19:58:17.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=149, tag_tid=11, entry_tid=33101' 2026-02-22T19:58:17.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:17.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:17.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:17.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=33101' 2026-02-22T19:58:17.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:17.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:17.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:17.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:17.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=33101' 2026-02-22T19:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33101 2026-02-22T19:58:17.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 33101 2026-02-22T19:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:17.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:17.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:17.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:17.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:17.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:17.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:17.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:17.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:17.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=33314], [object_number=149, tag_tid=11, entry_tid=33313], [object_number=148, tag_tid=11, entry_tid=33312], [object_number=151, tag_tid=11, entry_tid=33311]]], state=connected] 2026-02-22T19:58:17.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:17.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:17.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:17.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:17.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:17.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:17.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:17.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=33314], [object_number=149, tag_tid=11, entry_tid=33313], [object_number=148, tag_tid=11, entry_tid=33312], [object_number=151, tag_tid=11, entry_tid=33311]]], state=connected] 2026-02-22T19:58:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:17.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=150, tag_tid=11, entry_tid=33314' 2026-02-22T19:58:17.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=150, tag_tid=11, entry_tid=33314' 2026-02-22T19:58:17.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=33314' '!=' 'object_number=149, tag_tid=11, entry_tid=33101' 2026-02-22T19:58:17.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:17.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=33314' = 'object_number=149, tag_tid=11, entry_tid=33101' 2026-02-22T19:58:17.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=150, tag_tid=11, entry_tid=33314' 2026-02-22T19:58:17.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:17.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:17.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:17.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=33314' 2026-02-22T19:58:17.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:17.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:17.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:17.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:17.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=33314' 2026-02-22T19:58:17.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:17.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33314 2026-02-22T19:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 33314 2026-02-22T19:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:18.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:18.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:18.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:18.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:18.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:18.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:18.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:18.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:18.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:18.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:18.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:18.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:18.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:18.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:18.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:18.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:18.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:18.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:18.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:18.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:18.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:18.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:18.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:18.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=33526], [object_number=149, tag_tid=11, entry_tid=33525], [object_number=148, tag_tid=11, entry_tid=33524], [object_number=151, tag_tid=11, entry_tid=33523]]], state=connected] 2026-02-22T19:58:18.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:18.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:18.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:18.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:18.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:18.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:18.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:18.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=150, tag_tid=11, entry_tid=33526], [object_number=149, tag_tid=11, entry_tid=33525], [object_number=148, tag_tid=11, entry_tid=33524], [object_number=151, tag_tid=11, entry_tid=33523]]], state=connected] 2026-02-22T19:58:18.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=150, tag_tid=11, entry_tid=33526' 2026-02-22T19:58:18.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=150, tag_tid=11, entry_tid=33526' 2026-02-22T19:58:18.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=33526' '!=' 'object_number=150, tag_tid=11, entry_tid=33314' 2026-02-22T19:58:18.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:18.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=150, tag_tid=11, entry_tid=33526' = 'object_number=150, tag_tid=11, entry_tid=33314' 2026-02-22T19:58:18.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=150, tag_tid=11, entry_tid=33526' 2026-02-22T19:58:18.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:18.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:18.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:18.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:18.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=33526' 2026-02-22T19:58:18.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:18.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:18.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:18.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:18.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:18.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:18.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:18.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=150, tag_tid=11, entry_tid=33526' 2026-02-22T19:58:18.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:18.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33526 2026-02-22T19:58:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 33526 2026-02-22T19:58:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:18.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:18.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:18.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:18.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:18.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:18.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:18.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:18.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:18.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:18.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:18.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:18.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:18.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:18.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:18.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:18.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:18.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:18.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:18.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:18.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:18.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:18.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:18.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:18.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:18.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=33771], [object_number=150, tag_tid=11, entry_tid=33770], [object_number=149, tag_tid=11, entry_tid=33769], [object_number=148, tag_tid=11, entry_tid=33768]]], state=connected] 2026-02-22T19:58:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:18.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:18.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:18.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:18.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:18.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:18.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=33771], [object_number=150, tag_tid=11, entry_tid=33770], [object_number=149, tag_tid=11, entry_tid=33769], [object_number=148, tag_tid=11, entry_tid=33768]]], state=connected] 2026-02-22T19:58:18.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=151, tag_tid=11, entry_tid=33771' 2026-02-22T19:58:18.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=151, tag_tid=11, entry_tid=33771' 2026-02-22T19:58:18.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=33771' '!=' 'object_number=150, tag_tid=11, entry_tid=33526' 2026-02-22T19:58:18.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:18.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=33771' = 'object_number=150, tag_tid=11, entry_tid=33526' 2026-02-22T19:58:18.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=151, tag_tid=11, entry_tid=33771' 2026-02-22T19:58:18.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:18.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:18.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:18.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:18.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=33771' 2026-02-22T19:58:18.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:18.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:18.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:18.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:18.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:18.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:18.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:18.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=33771' 2026-02-22T19:58:18.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:18.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:18.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33771 2026-02-22T19:58:18.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 33771 2026-02-22T19:58:18.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:18.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:18.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:18.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:18.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:18.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:18.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:18.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:18.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:18.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:18.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:18.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:18.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:18.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:18.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:18.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:18.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:18.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:18.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:18.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:18.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:18.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=33983], [object_number=150, tag_tid=11, entry_tid=33982], [object_number=149, tag_tid=11, entry_tid=33981], [object_number=148, tag_tid=11, entry_tid=33980]]], state=connected] 2026-02-22T19:58:18.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:18.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:18.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:18.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:19.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:19.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:19.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:19.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=33983], [object_number=150, tag_tid=11, entry_tid=33982], [object_number=149, tag_tid=11, entry_tid=33981], [object_number=148, tag_tid=11, entry_tid=33980]]], state=connected] 2026-02-22T19:58:19.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=151, tag_tid=11, entry_tid=33983' 2026-02-22T19:58:19.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=151, tag_tid=11, entry_tid=33983' 2026-02-22T19:58:19.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=33983' '!=' 'object_number=151, tag_tid=11, entry_tid=33771' 2026-02-22T19:58:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=33983' = 'object_number=151, tag_tid=11, entry_tid=33771' 2026-02-22T19:58:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=151, tag_tid=11, entry_tid=33983' 2026-02-22T19:58:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:19.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:19.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:19.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=33983' 2026-02-22T19:58:19.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:19.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=33983' 2026-02-22T19:58:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33983 2026-02-22T19:58:19.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 33983 2026-02-22T19:58:19.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:19.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:19.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:19.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:19.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:19.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:19.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:19.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:19.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:19.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:19.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:19.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:19.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:19.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:19.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:19.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:19.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:19.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:19.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:19.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:19.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:19.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:19.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:19.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:19.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:19.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:19.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:19.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:19.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=34205], [object_number=148, tag_tid=11, entry_tid=34204], [object_number=151, tag_tid=11, entry_tid=34203], [object_number=150, tag_tid=11, entry_tid=34202]]], state=connected] 2026-02-22T19:58:19.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:19.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:19.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:19.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:19.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:19.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:19.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:19.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:19.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:19.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:19.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:19.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:19.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:19.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:19.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=34205], [object_number=148, tag_tid=11, entry_tid=34204], [object_number=151, tag_tid=11, entry_tid=34203], [object_number=150, tag_tid=11, entry_tid=34202]]], state=connected] 2026-02-22T19:58:19.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=149, tag_tid=11, entry_tid=34205' 2026-02-22T19:58:19.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=149, tag_tid=11, entry_tid=34205' 2026-02-22T19:58:19.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=34205' '!=' 'object_number=151, tag_tid=11, entry_tid=33983' 2026-02-22T19:58:19.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:19.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=34205' = 'object_number=151, tag_tid=11, entry_tid=33983' 2026-02-22T19:58:19.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=149, tag_tid=11, entry_tid=34205' 2026-02-22T19:58:19.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:19.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:19.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=34205' 2026-02-22T19:58:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=34205' 2026-02-22T19:58:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=34205 2026-02-22T19:58:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 34205 2026-02-22T19:58:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:19.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:19.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:19.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:19.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:19.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:19.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:19.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:19.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:19.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:19.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:19.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:19.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:19.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:19.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:19.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:19.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:19.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:19.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:19.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:19.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:19.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:19.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:19.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:19.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:19.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:19.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:19.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:19.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:19.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:19.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:19.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:19.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:19.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:19.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:19.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:19.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:19.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:19.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:19.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:19.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:19.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:19.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:19.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:19.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:19.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:19.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:19.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:19.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:19.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=34439], [object_number=150, tag_tid=11, entry_tid=34438], [object_number=149, tag_tid=11, entry_tid=34437], [object_number=148, tag_tid=11, entry_tid=34436]]], state=connected] 2026-02-22T19:58:19.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:19.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:19.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:19.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:19.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:19.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:19.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:19.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:19.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:19.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=34439], [object_number=150, tag_tid=11, entry_tid=34438], [object_number=149, tag_tid=11, entry_tid=34437], [object_number=148, tag_tid=11, entry_tid=34436]]], state=connected] 2026-02-22T19:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:19.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=151, tag_tid=11, entry_tid=34439' 2026-02-22T19:58:19.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=151, tag_tid=11, entry_tid=34439' 2026-02-22T19:58:19.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=34439' '!=' 'object_number=149, tag_tid=11, entry_tid=34205' 2026-02-22T19:58:19.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:19.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=34439' = 'object_number=149, tag_tid=11, entry_tid=34205' 2026-02-22T19:58:19.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=151, tag_tid=11, entry_tid=34439' 2026-02-22T19:58:19.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:19.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:19.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=34439' 2026-02-22T19:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:19.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:19.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=34439' 2026-02-22T19:58:19.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:19.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:19.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=34439 2026-02-22T19:58:19.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 34439 2026-02-22T19:58:19.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:19.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:19.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:19.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:19.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:19.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:19.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:19.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:19.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:19.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:19.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:20.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:20.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:20.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:20.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:20.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:20.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:20.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:20.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:20.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=34631], [object_number=150, tag_tid=11, entry_tid=34630], [object_number=149, tag_tid=11, entry_tid=34629], [object_number=148, tag_tid=11, entry_tid=34628]]], state=connected] 2026-02-22T19:58:20.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:20.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:20.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:20.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:20.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:20.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:20.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:20.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:20.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:20.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=34631], [object_number=150, tag_tid=11, entry_tid=34630], [object_number=149, tag_tid=11, entry_tid=34629], [object_number=148, tag_tid=11, entry_tid=34628]]], state=connected] 2026-02-22T19:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=151, tag_tid=11, entry_tid=34631' 2026-02-22T19:58:20.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=151, tag_tid=11, entry_tid=34631' 2026-02-22T19:58:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=34631' '!=' 'object_number=151, tag_tid=11, entry_tid=34439' 2026-02-22T19:58:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=34631' = 'object_number=151, tag_tid=11, entry_tid=34439' 2026-02-22T19:58:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=151, tag_tid=11, entry_tid=34631' 2026-02-22T19:58:20.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:20.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:20.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:20.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:20.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=34631' 2026-02-22T19:58:20.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:20.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:20.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=34631' 2026-02-22T19:58:20.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:20.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=34631 2026-02-22T19:58:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 34631 2026-02-22T19:58:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:20.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:20.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:20.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:20.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:20.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:20.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:20.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:20.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:20.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:20.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:20.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:20.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:20.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:20.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:20.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:20.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:20.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:20.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:20.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:20.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:20.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:20.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=34839], [object_number=150, tag_tid=11, entry_tid=34838], [object_number=149, tag_tid=11, entry_tid=34837], [object_number=148, tag_tid=11, entry_tid=34836]]], state=connected] 2026-02-22T19:58:20.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:20.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:20.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:20.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:20.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:20.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:20.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:20.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:20.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:20.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:20.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:20.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:20.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=151, tag_tid=11, entry_tid=34839], [object_number=150, tag_tid=11, entry_tid=34838], [object_number=149, tag_tid=11, entry_tid=34837], [object_number=148, tag_tid=11, entry_tid=34836]]], state=connected] 2026-02-22T19:58:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=151, tag_tid=11, entry_tid=34839' 2026-02-22T19:58:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=151, tag_tid=11, entry_tid=34839' 2026-02-22T19:58:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=34839' '!=' 'object_number=151, tag_tid=11, entry_tid=34631' 2026-02-22T19:58:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=151, tag_tid=11, entry_tid=34839' = 'object_number=151, tag_tid=11, entry_tid=34631' 2026-02-22T19:58:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=151, tag_tid=11, entry_tid=34839' 2026-02-22T19:58:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:20.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:20.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=34839' 2026-02-22T19:58:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=151, tag_tid=11, entry_tid=34839' 2026-02-22T19:58:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:20.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=34839 2026-02-22T19:58:20.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 34839 2026-02-22T19:58:20.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:20.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:20.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:20.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:20.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:20.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:20.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:20.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:20.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:20.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:20.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:20.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:20.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:20.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:20.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:20.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:20.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:20.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:20.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:20.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:20.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:20.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:20.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:20.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:20.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:20.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=148, tag_tid=11, entry_tid=35052], [object_number=151, tag_tid=11, entry_tid=35051], [object_number=150, tag_tid=11, entry_tid=35050], [object_number=149, tag_tid=11, entry_tid=35049]]], state=connected] 2026-02-22T19:58:20.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:20.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:20.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:20.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:20.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:20.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=148, tag_tid=11, entry_tid=35052], [object_number=151, tag_tid=11, entry_tid=35051], [object_number=150, tag_tid=11, entry_tid=35050], [object_number=149, tag_tid=11, entry_tid=35049]]], state=connected] 2026-02-22T19:58:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=148, tag_tid=11, entry_tid=35052' 2026-02-22T19:58:21.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=148, tag_tid=11, entry_tid=35052' 2026-02-22T19:58:21.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=148, tag_tid=11, entry_tid=35052' '!=' 'object_number=151, tag_tid=11, entry_tid=34839' 2026-02-22T19:58:21.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:21.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=148, tag_tid=11, entry_tid=35052' = 'object_number=151, tag_tid=11, entry_tid=34839' 2026-02-22T19:58:21.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=148, tag_tid=11, entry_tid=35052' 2026-02-22T19:58:21.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:21.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:21.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:21.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=148, tag_tid=11, entry_tid=35052' 2026-02-22T19:58:21.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:21.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:21.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:21.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=148, tag_tid=11, entry_tid=35052' 2026-02-22T19:58:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:21.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35052 2026-02-22T19:58:21.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 35052 2026-02-22T19:58:21.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:21.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:21.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:21.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:21.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:21.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:21.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:21.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:21.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:21.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:21.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:21.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:21.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:21.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:21.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:21.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:21.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:21.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:21.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:21.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:21.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:21.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=35305], [object_number=148, tag_tid=11, entry_tid=35304], [object_number=151, tag_tid=11, entry_tid=35303], [object_number=150, tag_tid=11, entry_tid=35302]]], state=connected] 2026-02-22T19:58:21.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:21.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:21.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:21.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:21.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:21.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:21.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:21.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:21.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:21.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:21.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:21.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:21.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:21.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:21.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:21.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:21.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:21.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:21.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=35305], [object_number=148, tag_tid=11, entry_tid=35304], [object_number=151, tag_tid=11, entry_tid=35303], [object_number=150, tag_tid=11, entry_tid=35302]]], state=connected] 2026-02-22T19:58:21.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=149, tag_tid=11, entry_tid=35305' 2026-02-22T19:58:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=149, tag_tid=11, entry_tid=35305' 2026-02-22T19:58:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=35305' '!=' 'object_number=148, tag_tid=11, entry_tid=35052' 2026-02-22T19:58:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=35305' = 'object_number=148, tag_tid=11, entry_tid=35052' 2026-02-22T19:58:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=149, tag_tid=11, entry_tid=35305' 2026-02-22T19:58:21.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:21.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:21.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:21.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=35305' 2026-02-22T19:58:21.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:21.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:21.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:21.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:21.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:21.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=35305' 2026-02-22T19:58:21.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:21.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35305 2026-02-22T19:58:21.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 35305 2026-02-22T19:58:21.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:21.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:21.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:21.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:21.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:21.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:21.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:21.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:21.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:21.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:21.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:21.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:21.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:21.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:21.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:21.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:21.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:21.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:21.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:21.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:21.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:21.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:21.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:21.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:21.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:21.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:21.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:21.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:21.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:21.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:21.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:21.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:21.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=35529], [object_number=148, tag_tid=11, entry_tid=35528], [object_number=151, tag_tid=11, entry_tid=35527], [object_number=150, tag_tid=11, entry_tid=35526]]], state=connected] 2026-02-22T19:58:21.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:21.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 37 2026-02-22T19:58:21.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:21.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:21.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:21.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=149, tag_tid=11, entry_tid=35529], [object_number=148, tag_tid=11, entry_tid=35528], [object_number=151, tag_tid=11, entry_tid=35527], [object_number=150, tag_tid=11, entry_tid=35526]]], state=connected] 2026-02-22T19:58:21.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=149, tag_tid=11, entry_tid=35529' 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=149, tag_tid=11, entry_tid=35529' 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=35529' '!=' 'object_number=149, tag_tid=11, entry_tid=35305' 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=149, tag_tid=11, entry_tid=35529' = 'object_number=149, tag_tid=11, entry_tid=35305' 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=149, tag_tid=11, entry_tid=35529' 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:21.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:21.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=35529' 2026-02-22T19:58:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:21.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:21.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:21.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:21.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=149, tag_tid=11, entry_tid=35529' 2026-02-22T19:58:21.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:21.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:21.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35529 2026-02-22T19:58:21.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 35529 2026-02-22T19:58:21.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:21.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:21.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:22.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:22.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:22.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:22.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:22.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:22.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:22.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:22.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:22.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:22.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:22.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:22.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:22.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:22.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:22.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:22.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:22.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:22.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:22.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:22.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:22.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:22.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:22.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:22.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:22.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:22.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:22.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:22.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:22.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:22.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:22.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:22.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:22.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:22.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:22.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:22.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:22.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=155, tag_tid=11, entry_tid=35679], [object_number=154, tag_tid=11, entry_tid=35678], [object_number=153, tag_tid=11, entry_tid=35677], [object_number=152, tag_tid=11, entry_tid=35676]]], state=connected] 2026-02-22T19:58:22.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:22.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:22.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:22.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:22.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:22.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:22.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=155, tag_tid=11, entry_tid=35679], [object_number=154, tag_tid=11, entry_tid=35678], [object_number=153, tag_tid=11, entry_tid=35677], [object_number=152, tag_tid=11, entry_tid=35676]]], state=connected] 2026-02-22T19:58:22.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=155, tag_tid=11, entry_tid=35679' 2026-02-22T19:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=155, tag_tid=11, entry_tid=35679' 2026-02-22T19:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=155, tag_tid=11, entry_tid=35679' '!=' 'object_number=149, tag_tid=11, entry_tid=35529' 2026-02-22T19:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=155, tag_tid=11, entry_tid=35679' = 'object_number=149, tag_tid=11, entry_tid=35529' 2026-02-22T19:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=155, tag_tid=11, entry_tid=35679' 2026-02-22T19:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:22.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:22.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=155, tag_tid=11, entry_tid=35679' 2026-02-22T19:58:22.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:22.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:22.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:22.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:22.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:22.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:22.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:22.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=155, tag_tid=11, entry_tid=35679' 2026-02-22T19:58:22.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:22.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:22.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35679 2026-02-22T19:58:22.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 35679 2026-02-22T19:58:22.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:22.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:22.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:22.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:22.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:22.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:22.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:22.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:22.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:22.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:22.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:22.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:22.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:22.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:22.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:22.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=35910], [object_number=153, tag_tid=11, entry_tid=35909], [object_number=152, tag_tid=11, entry_tid=35908], [object_number=155, tag_tid=11, entry_tid=35907]]], state=connected] 2026-02-22T19:58:22.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:22.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:22.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:22.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:22.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:22.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:22.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=35910], [object_number=153, tag_tid=11, entry_tid=35909], [object_number=152, tag_tid=11, entry_tid=35908], [object_number=155, tag_tid=11, entry_tid=35907]]], state=connected] 2026-02-22T19:58:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=154, tag_tid=11, entry_tid=35910' 2026-02-22T19:58:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=154, tag_tid=11, entry_tid=35910' 2026-02-22T19:58:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=35910' '!=' 'object_number=155, tag_tid=11, entry_tid=35679' 2026-02-22T19:58:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=35910' = 'object_number=155, tag_tid=11, entry_tid=35679' 2026-02-22T19:58:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=154, tag_tid=11, entry_tid=35910' 2026-02-22T19:58:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=35910' 2026-02-22T19:58:22.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:22.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:22.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=35910' 2026-02-22T19:58:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35910 2026-02-22T19:58:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 35910 2026-02-22T19:58:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:22.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:22.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:22.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:22.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:22.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=153, tag_tid=11, entry_tid=36129], [object_number=152, tag_tid=11, entry_tid=36128], [object_number=155, tag_tid=11, entry_tid=36127], [object_number=154, tag_tid=11, entry_tid=36126]]], state=connected] 2026-02-22T19:58:22.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:22.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:22.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:22.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:22.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:22.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:22.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:23.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:23.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:23.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:23.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=153, tag_tid=11, entry_tid=36129], [object_number=152, tag_tid=11, entry_tid=36128], [object_number=155, tag_tid=11, entry_tid=36127], [object_number=154, tag_tid=11, entry_tid=36126]]], state=connected] 2026-02-22T19:58:23.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=153, tag_tid=11, entry_tid=36129' 2026-02-22T19:58:23.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=153, tag_tid=11, entry_tid=36129' 2026-02-22T19:58:23.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=153, tag_tid=11, entry_tid=36129' '!=' 'object_number=154, tag_tid=11, entry_tid=35910' 2026-02-22T19:58:23.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:23.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=153, tag_tid=11, entry_tid=36129' = 'object_number=154, tag_tid=11, entry_tid=35910' 2026-02-22T19:58:23.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=153, tag_tid=11, entry_tid=36129' 2026-02-22T19:58:23.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:23.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:23.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:23.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=153, tag_tid=11, entry_tid=36129' 2026-02-22T19:58:23.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:23.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=153, tag_tid=11, entry_tid=36129' 2026-02-22T19:58:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36129 2026-02-22T19:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 36129 2026-02-22T19:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:23.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:23.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:23.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:23.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:23.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:23.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:23.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:23.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:23.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:23.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:23.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:23.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:23.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:23.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:23.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:23.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:23.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:23.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:23.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:23.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:23.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=36330], [object_number=153, tag_tid=11, entry_tid=36329], [object_number=152, tag_tid=11, entry_tid=36328], [object_number=155, tag_tid=11, entry_tid=36327]]], state=connected] 2026-02-22T19:58:23.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:23.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:23.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:23.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:23.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:23.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:23.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:23.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:23.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:23.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=36330], [object_number=153, tag_tid=11, entry_tid=36329], [object_number=152, tag_tid=11, entry_tid=36328], [object_number=155, tag_tid=11, entry_tid=36327]]], state=connected] 2026-02-22T19:58:23.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=154, tag_tid=11, entry_tid=36330' 2026-02-22T19:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=154, tag_tid=11, entry_tid=36330' 2026-02-22T19:58:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=36330' '!=' 'object_number=153, tag_tid=11, entry_tid=36129' 2026-02-22T19:58:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=36330' = 'object_number=153, tag_tid=11, entry_tid=36129' 2026-02-22T19:58:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=154, tag_tid=11, entry_tid=36330' 2026-02-22T19:58:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:23.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=36330' 2026-02-22T19:58:23.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:23.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:23.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:23.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:23.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=36330' 2026-02-22T19:58:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36330 2026-02-22T19:58:23.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 36330 2026-02-22T19:58:23.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:23.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:23.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:23.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:23.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:23.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:23.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:23.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:23.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:23.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:23.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:23.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:23.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:23.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:23.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:23.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:23.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:23.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:23.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:23.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:23.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:23.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:23.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:23.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:23.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:23.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:23.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:23.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:23.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:23.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:23.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:23.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:23.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:23.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:23.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:23.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:23.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=152, tag_tid=11, entry_tid=36516], [object_number=155, tag_tid=11, entry_tid=36515], [object_number=154, tag_tid=11, entry_tid=36514], [object_number=153, tag_tid=11, entry_tid=36513]]], state=connected] 2026-02-22T19:58:23.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:23.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:23.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:23.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:23.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:23.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:23.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:23.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:23.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:23.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:23.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:23.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:23.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:23.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:23.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:23.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:23.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:23.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:23.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=152, tag_tid=11, entry_tid=36516], [object_number=155, tag_tid=11, entry_tid=36515], [object_number=154, tag_tid=11, entry_tid=36514], [object_number=153, tag_tid=11, entry_tid=36513]]], state=connected] 2026-02-22T19:58:23.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=152, tag_tid=11, entry_tid=36516' 2026-02-22T19:58:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=152, tag_tid=11, entry_tid=36516' 2026-02-22T19:58:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=152, tag_tid=11, entry_tid=36516' '!=' 'object_number=154, tag_tid=11, entry_tid=36330' 2026-02-22T19:58:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=152, tag_tid=11, entry_tid=36516' = 'object_number=154, tag_tid=11, entry_tid=36330' 2026-02-22T19:58:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=152, tag_tid=11, entry_tid=36516' 2026-02-22T19:58:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:23.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=152, tag_tid=11, entry_tid=36516' 2026-02-22T19:58:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:23.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:23.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:23.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=152, tag_tid=11, entry_tid=36516' 2026-02-22T19:58:23.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:23.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36516 2026-02-22T19:58:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 36516 2026-02-22T19:58:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:23.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:23.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:23.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:23.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:23.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:23.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:23.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:23.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:24.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:24.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:24.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:24.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:24.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:24.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:24.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:24.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:24.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=36734], [object_number=153, tag_tid=11, entry_tid=36733], [object_number=152, tag_tid=11, entry_tid=36732], [object_number=155, tag_tid=11, entry_tid=36731]]], state=connected] 2026-02-22T19:58:24.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:24.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:24.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:24.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:24.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:24.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:24.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:24.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:24.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:24.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=36734], [object_number=153, tag_tid=11, entry_tid=36733], [object_number=152, tag_tid=11, entry_tid=36732], [object_number=155, tag_tid=11, entry_tid=36731]]], state=connected] 2026-02-22T19:58:24.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=154, tag_tid=11, entry_tid=36734' 2026-02-22T19:58:24.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=154, tag_tid=11, entry_tid=36734' 2026-02-22T19:58:24.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=36734' '!=' 'object_number=152, tag_tid=11, entry_tid=36516' 2026-02-22T19:58:24.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:24.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=36734' = 'object_number=152, tag_tid=11, entry_tid=36516' 2026-02-22T19:58:24.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=154, tag_tid=11, entry_tid=36734' 2026-02-22T19:58:24.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:24.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:24.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:24.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:24.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=36734' 2026-02-22T19:58:24.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:24.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:24.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:24.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:24.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:24.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:24.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:24.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=36734' 2026-02-22T19:58:24.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:24.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:24.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36734 2026-02-22T19:58:24.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 36734 2026-02-22T19:58:24.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:24.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:24.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:24.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:24.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:24.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:24.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:24.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:24.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:24.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:24.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:24.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:24.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:24.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:24.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:24.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=152, tag_tid=11, entry_tid=37116], [object_number=155, tag_tid=11, entry_tid=37115], [object_number=154, tag_tid=11, entry_tid=37114], [object_number=153, tag_tid=11, entry_tid=37113]]], state=connected] 2026-02-22T19:58:24.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:24.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:24.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:24.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:24.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:24.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:24.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:24.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:24.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:24.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:24.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:24.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:24.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:24.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:24.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=152, tag_tid=11, entry_tid=37116], [object_number=155, tag_tid=11, entry_tid=37115], [object_number=154, tag_tid=11, entry_tid=37114], [object_number=153, tag_tid=11, entry_tid=37113]]], state=connected] 2026-02-22T19:58:24.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=152, tag_tid=11, entry_tid=37116' 2026-02-22T19:58:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=152, tag_tid=11, entry_tid=37116' 2026-02-22T19:58:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=152, tag_tid=11, entry_tid=37116' '!=' 'object_number=154, tag_tid=11, entry_tid=36734' 2026-02-22T19:58:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=152, tag_tid=11, entry_tid=37116' = 'object_number=154, tag_tid=11, entry_tid=36734' 2026-02-22T19:58:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=152, tag_tid=11, entry_tid=37116' 2026-02-22T19:58:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:24.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:24.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=152, tag_tid=11, entry_tid=37116' 2026-02-22T19:58:24.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:24.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:24.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:24.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:24.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:24.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=152, tag_tid=11, entry_tid=37116' 2026-02-22T19:58:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37116 2026-02-22T19:58:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 37116 2026-02-22T19:58:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:24.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:24.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:24.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:24.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:25.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:25.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:25.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:25.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:25.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:25.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:25.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:25.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=37354], [object_number=153, tag_tid=11, entry_tid=37353], [object_number=152, tag_tid=11, entry_tid=37352], [object_number=155, tag_tid=11, entry_tid=37351]]], state=connected] 2026-02-22T19:58:25.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:25.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:25.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:25.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=37354], [object_number=153, tag_tid=11, entry_tid=37353], [object_number=152, tag_tid=11, entry_tid=37352], [object_number=155, tag_tid=11, entry_tid=37351]]], state=connected] 2026-02-22T19:58:25.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=154, tag_tid=11, entry_tid=37354' 2026-02-22T19:58:25.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=154, tag_tid=11, entry_tid=37354' 2026-02-22T19:58:25.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=37354' '!=' 'object_number=152, tag_tid=11, entry_tid=37116' 2026-02-22T19:58:25.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:25.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=37354' = 'object_number=152, tag_tid=11, entry_tid=37116' 2026-02-22T19:58:25.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=154, tag_tid=11, entry_tid=37354' 2026-02-22T19:58:25.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:25.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:25.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:25.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=37354' 2026-02-22T19:58:25.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:25.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:25.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:25.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:25.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:25.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=37354' 2026-02-22T19:58:25.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:25.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37354 2026-02-22T19:58:25.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 37354 2026-02-22T19:58:25.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:25.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:25.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:25.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:25.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:25.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:25.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:25.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:25.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:25.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:25.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:25.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:25.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:25.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:25.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:25.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:25.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:25.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:25.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:25.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:25.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=153, tag_tid=11, entry_tid=37553], [object_number=152, tag_tid=11, entry_tid=37552], [object_number=155, tag_tid=11, entry_tid=37551], [object_number=154, tag_tid=11, entry_tid=37550]]], state=connected] 2026-02-22T19:58:25.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:25.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:25.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:25.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:25.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:25.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:25.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:25.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:25.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:25.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:25.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:25.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:25.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=153, tag_tid=11, entry_tid=37553], [object_number=152, tag_tid=11, entry_tid=37552], [object_number=155, tag_tid=11, entry_tid=37551], [object_number=154, tag_tid=11, entry_tid=37550]]], state=connected] 2026-02-22T19:58:25.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=153, tag_tid=11, entry_tid=37553' 2026-02-22T19:58:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=153, tag_tid=11, entry_tid=37553' 2026-02-22T19:58:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=153, tag_tid=11, entry_tid=37553' '!=' 'object_number=154, tag_tid=11, entry_tid=37354' 2026-02-22T19:58:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=153, tag_tid=11, entry_tid=37553' = 'object_number=154, tag_tid=11, entry_tid=37354' 2026-02-22T19:58:25.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=153, tag_tid=11, entry_tid=37553' 2026-02-22T19:58:25.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:25.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:25.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:25.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=153, tag_tid=11, entry_tid=37553' 2026-02-22T19:58:25.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:25.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:25.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:25.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:25.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:25.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=153, tag_tid=11, entry_tid=37553' 2026-02-22T19:58:25.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:25.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37553 2026-02-22T19:58:25.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 37553 2026-02-22T19:58:25.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:25.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:25.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:25.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:25.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:25.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:25.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:25.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:25.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:25.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:25.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:25.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:25.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:25.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:25.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:25.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:25.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:25.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:25.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:25.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:25.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:25.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:25.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:25.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:25.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:25.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:25.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:25.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:25.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:25.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:25.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:25.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:25.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:25.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:25.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:25.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:25.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:25.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:25.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:25.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:25.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:25.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:25.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:25.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:25.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=37738], [object_number=153, tag_tid=11, entry_tid=37737], [object_number=152, tag_tid=11, entry_tid=37736], [object_number=155, tag_tid=11, entry_tid=37735]]], state=connected] 2026-02-22T19:58:25.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:25.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=37738], [object_number=153, tag_tid=11, entry_tid=37737], [object_number=152, tag_tid=11, entry_tid=37736], [object_number=155, tag_tid=11, entry_tid=37735]]], state=connected] 2026-02-22T19:58:25.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:25.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=154, tag_tid=11, entry_tid=37738' 2026-02-22T19:58:25.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=154, tag_tid=11, entry_tid=37738' 2026-02-22T19:58:25.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=37738' '!=' 'object_number=153, tag_tid=11, entry_tid=37553' 2026-02-22T19:58:25.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:25.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=37738' = 'object_number=153, tag_tid=11, entry_tid=37553' 2026-02-22T19:58:25.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=154, tag_tid=11, entry_tid=37738' 2026-02-22T19:58:25.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:25.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:25.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:25.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=37738' 2026-02-22T19:58:25.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:25.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:25.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:25.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:25.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:25.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=37738' 2026-02-22T19:58:25.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:25.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:25.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37738 2026-02-22T19:58:25.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 37738 2026-02-22T19:58:25.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:25.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:25.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:25.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:25.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:25.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:25.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:26.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:26.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:26.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:26.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:26.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:26.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:26.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=152, tag_tid=11, entry_tid=37956], [object_number=155, tag_tid=11, entry_tid=37955], [object_number=154, tag_tid=11, entry_tid=37954], [object_number=153, tag_tid=11, entry_tid=37953]]], state=connected] 2026-02-22T19:58:26.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=152, tag_tid=11, entry_tid=37956], [object_number=155, tag_tid=11, entry_tid=37955], [object_number=154, tag_tid=11, entry_tid=37954], [object_number=153, tag_tid=11, entry_tid=37953]]], state=connected] 2026-02-22T19:58:26.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=152, tag_tid=11, entry_tid=37956' 2026-02-22T19:58:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=152, tag_tid=11, entry_tid=37956' 2026-02-22T19:58:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=152, tag_tid=11, entry_tid=37956' '!=' 'object_number=154, tag_tid=11, entry_tid=37738' 2026-02-22T19:58:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=152, tag_tid=11, entry_tid=37956' = 'object_number=154, tag_tid=11, entry_tid=37738' 2026-02-22T19:58:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=152, tag_tid=11, entry_tid=37956' 2026-02-22T19:58:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:26.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:26.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:26.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=152, tag_tid=11, entry_tid=37956' 2026-02-22T19:58:26.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:26.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:26.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:26.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:26.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:26.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=152, tag_tid=11, entry_tid=37956' 2026-02-22T19:58:26.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:26.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37956 2026-02-22T19:58:26.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 37956 2026-02-22T19:58:26.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:26.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:26.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:26.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:26.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:26.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:26.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:26.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:26.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:26.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:26.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:26.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:26.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:26.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:26.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:26.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:26.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:26.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:26.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:26.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:26.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=155, tag_tid=11, entry_tid=38175], [object_number=154, tag_tid=11, entry_tid=38174], [object_number=153, tag_tid=11, entry_tid=38173], [object_number=152, tag_tid=11, entry_tid=38172]]], state=connected] 2026-02-22T19:58:26.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:26.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=155, tag_tid=11, entry_tid=38175], [object_number=154, tag_tid=11, entry_tid=38174], [object_number=153, tag_tid=11, entry_tid=38173], [object_number=152, tag_tid=11, entry_tid=38172]]], state=connected] 2026-02-22T19:58:26.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=155, tag_tid=11, entry_tid=38175' 2026-02-22T19:58:26.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=155, tag_tid=11, entry_tid=38175' 2026-02-22T19:58:26.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=155, tag_tid=11, entry_tid=38175' '!=' 'object_number=152, tag_tid=11, entry_tid=37956' 2026-02-22T19:58:26.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:26.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=155, tag_tid=11, entry_tid=38175' = 'object_number=152, tag_tid=11, entry_tid=37956' 2026-02-22T19:58:26.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=155, tag_tid=11, entry_tid=38175' 2026-02-22T19:58:26.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:26.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:26.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:26.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=155, tag_tid=11, entry_tid=38175' 2026-02-22T19:58:26.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:26.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:26.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:26.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:26.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:26.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=155, tag_tid=11, entry_tid=38175' 2026-02-22T19:58:26.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:26.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38175 2026-02-22T19:58:26.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 38175 2026-02-22T19:58:26.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:26.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:26.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:26.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:26.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:26.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:26.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:26.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:26.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:26.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:26.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:26.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:26.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:26.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:26.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:26.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:26.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:26.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:26.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:26.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:26.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:26.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:26.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=38382], [object_number=153, tag_tid=11, entry_tid=38381], [object_number=152, tag_tid=11, entry_tid=38380], [object_number=155, tag_tid=11, entry_tid=38379]]], state=connected] 2026-02-22T19:58:26.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:26.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:26.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:26.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:26.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:26.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=38382], [object_number=153, tag_tid=11, entry_tid=38381], [object_number=152, tag_tid=11, entry_tid=38380], [object_number=155, tag_tid=11, entry_tid=38379]]], state=connected] 2026-02-22T19:58:26.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:26.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=154, tag_tid=11, entry_tid=38382' 2026-02-22T19:58:26.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=154, tag_tid=11, entry_tid=38382' 2026-02-22T19:58:26.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=38382' '!=' 'object_number=155, tag_tid=11, entry_tid=38175' 2026-02-22T19:58:26.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:26.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=38382' = 'object_number=155, tag_tid=11, entry_tid=38175' 2026-02-22T19:58:26.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=154, tag_tid=11, entry_tid=38382' 2026-02-22T19:58:26.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:26.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:26.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:26.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=38382' 2026-02-22T19:58:26.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:26.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=38382' 2026-02-22T19:58:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:26.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38382 2026-02-22T19:58:26.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 38382 2026-02-22T19:58:26.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:26.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:26.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:27.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:27.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:27.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:27.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:27.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:27.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:27.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:27.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:27.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:27.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:27.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:27.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:27.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:27.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:27.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:27.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:27.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:27.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:27.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:27.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:27.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:27.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:27.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=155, tag_tid=11, entry_tid=38587], [object_number=154, tag_tid=11, entry_tid=38586], [object_number=153, tag_tid=11, entry_tid=38585], [object_number=152, tag_tid=11, entry_tid=38584]]], state=connected] 2026-02-22T19:58:27.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:27.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:27.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:27.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:27.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:27.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=155, tag_tid=11, entry_tid=38587], [object_number=154, tag_tid=11, entry_tid=38586], [object_number=153, tag_tid=11, entry_tid=38585], [object_number=152, tag_tid=11, entry_tid=38584]]], state=connected] 2026-02-22T19:58:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=155, tag_tid=11, entry_tid=38587' 2026-02-22T19:58:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=155, tag_tid=11, entry_tid=38587' 2026-02-22T19:58:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=155, tag_tid=11, entry_tid=38587' '!=' 'object_number=154, tag_tid=11, entry_tid=38382' 2026-02-22T19:58:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=155, tag_tid=11, entry_tid=38587' = 'object_number=154, tag_tid=11, entry_tid=38382' 2026-02-22T19:58:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=155, tag_tid=11, entry_tid=38587' 2026-02-22T19:58:27.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:27.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:27.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:27.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=155, tag_tid=11, entry_tid=38587' 2026-02-22T19:58:27.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:27.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:27.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=155, tag_tid=11, entry_tid=38587' 2026-02-22T19:58:27.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:27.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38587 2026-02-22T19:58:27.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 38587 2026-02-22T19:58:27.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:27.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:27.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:27.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:27.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:27.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:27.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:27.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:27.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:27.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:27.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:27.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:27.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:27.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:27.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:27.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:27.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:27.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:27.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:27.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:27.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:27.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:27.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:27.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:27.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:27.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:27.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:27.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:27.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=38798], [object_number=153, tag_tid=11, entry_tid=38797], [object_number=152, tag_tid=11, entry_tid=38796], [object_number=155, tag_tid=11, entry_tid=38795]]], state=connected] 2026-02-22T19:58:27.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:27.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:27.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:27.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:27.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:27.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:27.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:27.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:27.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:27.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:27.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=38798], [object_number=153, tag_tid=11, entry_tid=38797], [object_number=152, tag_tid=11, entry_tid=38796], [object_number=155, tag_tid=11, entry_tid=38795]]], state=connected] 2026-02-22T19:58:27.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=154, tag_tid=11, entry_tid=38798' 2026-02-22T19:58:27.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=154, tag_tid=11, entry_tid=38798' 2026-02-22T19:58:27.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=38798' '!=' 'object_number=155, tag_tid=11, entry_tid=38587' 2026-02-22T19:58:27.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:27.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=38798' = 'object_number=155, tag_tid=11, entry_tid=38587' 2026-02-22T19:58:27.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=154, tag_tid=11, entry_tid=38798' 2026-02-22T19:58:27.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:27.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:27.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=38798' 2026-02-22T19:58:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:27.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:27.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:27.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:27.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=38798' 2026-02-22T19:58:27.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:27.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38798 2026-02-22T19:58:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 38798 2026-02-22T19:58:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:27.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:27.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:27.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:27.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:27.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:27.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:27.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:27.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:27.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:27.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:27.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:27.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:27.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:27.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:27.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:27.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:27.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:27.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:27.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:27.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:27.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=39018], [object_number=153, tag_tid=11, entry_tid=39017], [object_number=152, tag_tid=11, entry_tid=39016], [object_number=155, tag_tid=11, entry_tid=39015]]], state=connected] 2026-02-22T19:58:27.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:27.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=39018], [object_number=153, tag_tid=11, entry_tid=39017], [object_number=152, tag_tid=11, entry_tid=39016], [object_number=155, tag_tid=11, entry_tid=39015]]], state=connected] 2026-02-22T19:58:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=154, tag_tid=11, entry_tid=39018' 2026-02-22T19:58:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=154, tag_tid=11, entry_tid=39018' 2026-02-22T19:58:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=39018' '!=' 'object_number=154, tag_tid=11, entry_tid=38798' 2026-02-22T19:58:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=39018' = 'object_number=154, tag_tid=11, entry_tid=38798' 2026-02-22T19:58:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=154, tag_tid=11, entry_tid=39018' 2026-02-22T19:58:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:27.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=39018' 2026-02-22T19:58:27.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:27.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:27.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:27.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:27.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:27.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=39018' 2026-02-22T19:58:27.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:27.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:27.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39018 2026-02-22T19:58:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 39018 2026-02-22T19:58:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:28.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:28.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:28.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:28.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:28.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:28.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:28.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:28.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:28.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:28.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:28.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:28.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:28.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:28.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:28.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:28.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:28.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=153, tag_tid=11, entry_tid=39229], [object_number=152, tag_tid=11, entry_tid=39228], [object_number=155, tag_tid=11, entry_tid=39227], [object_number=154, tag_tid=11, entry_tid=39226]]], state=connected] 2026-02-22T19:58:28.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:28.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:28.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:28.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:28.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:28.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:28.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=153, tag_tid=11, entry_tid=39229], [object_number=152, tag_tid=11, entry_tid=39228], [object_number=155, tag_tid=11, entry_tid=39227], [object_number=154, tag_tid=11, entry_tid=39226]]], state=connected] 2026-02-22T19:58:28.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=153, tag_tid=11, entry_tid=39229' 2026-02-22T19:58:28.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=153, tag_tid=11, entry_tid=39229' 2026-02-22T19:58:28.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=153, tag_tid=11, entry_tid=39229' '!=' 'object_number=154, tag_tid=11, entry_tid=39018' 2026-02-22T19:58:28.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:28.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=153, tag_tid=11, entry_tid=39229' = 'object_number=154, tag_tid=11, entry_tid=39018' 2026-02-22T19:58:28.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=153, tag_tid=11, entry_tid=39229' 2026-02-22T19:58:28.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:28.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:28.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:28.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:28.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=153, tag_tid=11, entry_tid=39229' 2026-02-22T19:58:28.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:28.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:28.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:28.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:28.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:28.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:28.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:28.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:28.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=153, tag_tid=11, entry_tid=39229' 2026-02-22T19:58:28.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:28.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39229 2026-02-22T19:58:28.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 39229 2026-02-22T19:58:28.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:28.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:28.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:28.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:28.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:28.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:28.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:28.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:28.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:28.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:28.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:28.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:28.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:28.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:28.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:28.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:28.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:28.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:28.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:28.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:28.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:28.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=152, tag_tid=11, entry_tid=39440], [object_number=155, tag_tid=11, entry_tid=39439], [object_number=154, tag_tid=11, entry_tid=39438], [object_number=153, tag_tid=11, entry_tid=39437]]], state=connected] 2026-02-22T19:58:28.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:28.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:28.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:28.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:28.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:28.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:28.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:28.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:28.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=152, tag_tid=11, entry_tid=39440], [object_number=155, tag_tid=11, entry_tid=39439], [object_number=154, tag_tid=11, entry_tid=39438], [object_number=153, tag_tid=11, entry_tid=39437]]], state=connected] 2026-02-22T19:58:28.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=152, tag_tid=11, entry_tid=39440' 2026-02-22T19:58:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=152, tag_tid=11, entry_tid=39440' 2026-02-22T19:58:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=152, tag_tid=11, entry_tid=39440' '!=' 'object_number=153, tag_tid=11, entry_tid=39229' 2026-02-22T19:58:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=152, tag_tid=11, entry_tid=39440' = 'object_number=153, tag_tid=11, entry_tid=39229' 2026-02-22T19:58:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=152, tag_tid=11, entry_tid=39440' 2026-02-22T19:58:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:28.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:28.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:28.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=152, tag_tid=11, entry_tid=39440' 2026-02-22T19:58:28.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:28.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:28.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:28.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:28.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=152, tag_tid=11, entry_tid=39440' 2026-02-22T19:58:28.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:28.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:28.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39440 2026-02-22T19:58:28.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 39440 2026-02-22T19:58:28.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:28.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:28.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:28.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:28.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:28.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:28.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:28.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:28.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:28.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:28.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:28.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:28.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:28.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:28.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:28.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:28.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:28.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:29.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:29.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:29.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:29.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=39638], [object_number=153, tag_tid=11, entry_tid=39637], [object_number=152, tag_tid=11, entry_tid=39636], [object_number=155, tag_tid=11, entry_tid=39635]]], state=connected] 2026-02-22T19:58:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:29.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:29.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 38 2026-02-22T19:58:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=154, tag_tid=11, entry_tid=39638], [object_number=153, tag_tid=11, entry_tid=39637], [object_number=152, tag_tid=11, entry_tid=39636], [object_number=155, tag_tid=11, entry_tid=39635]]], state=connected] 2026-02-22T19:58:29.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=154, tag_tid=11, entry_tid=39638' 2026-02-22T19:58:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=154, tag_tid=11, entry_tid=39638' 2026-02-22T19:58:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=39638' '!=' 'object_number=152, tag_tid=11, entry_tid=39440' 2026-02-22T19:58:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=154, tag_tid=11, entry_tid=39638' = 'object_number=152, tag_tid=11, entry_tid=39440' 2026-02-22T19:58:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=154, tag_tid=11, entry_tid=39638' 2026-02-22T19:58:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:29.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=39638' 2026-02-22T19:58:29.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:29.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:29.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:29.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:29.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=154, tag_tid=11, entry_tid=39638' 2026-02-22T19:58:29.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:29.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39638 2026-02-22T19:58:29.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 39638 2026-02-22T19:58:29.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:29.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:29.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:29.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:29.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:29.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:29.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:29.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:29.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:29.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:29.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:29.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:29.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:29.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:29.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:29.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:29.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:29.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:29.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:29.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:29.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:29.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:29.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:29.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:29.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:29.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:29.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:29.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:29.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=159, tag_tid=11, entry_tid=39787], [object_number=158, tag_tid=11, entry_tid=39786], [object_number=157, tag_tid=11, entry_tid=39785], [object_number=156, tag_tid=11, entry_tid=39784]]], state=connected] 2026-02-22T19:58:29.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:29.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:29.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:29.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:29.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:29.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:29.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:29.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:29.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:29.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=159, tag_tid=11, entry_tid=39787], [object_number=158, tag_tid=11, entry_tid=39786], [object_number=157, tag_tid=11, entry_tid=39785], [object_number=156, tag_tid=11, entry_tid=39784]]], state=connected] 2026-02-22T19:58:29.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=159, tag_tid=11, entry_tid=39787' 2026-02-22T19:58:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=159, tag_tid=11, entry_tid=39787' 2026-02-22T19:58:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=159, tag_tid=11, entry_tid=39787' '!=' 'object_number=154, tag_tid=11, entry_tid=39638' 2026-02-22T19:58:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=159, tag_tid=11, entry_tid=39787' = 'object_number=154, tag_tid=11, entry_tid=39638' 2026-02-22T19:58:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=159, tag_tid=11, entry_tid=39787' 2026-02-22T19:58:29.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:29.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:29.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=159, tag_tid=11, entry_tid=39787' 2026-02-22T19:58:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:29.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:29.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:29.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=159, tag_tid=11, entry_tid=39787' 2026-02-22T19:58:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39787 2026-02-22T19:58:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 39787 2026-02-22T19:58:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:29.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:29.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:29.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:29.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:29.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:29.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:29.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:29.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:29.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:29.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:29.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:29.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:29.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:29.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:29.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:29.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=159, tag_tid=11, entry_tid=40011], [object_number=158, tag_tid=11, entry_tid=40010], [object_number=157, tag_tid=11, entry_tid=40009], [object_number=156, tag_tid=11, entry_tid=40008]]], state=connected] 2026-02-22T19:58:29.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:29.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:29.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:29.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:29.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=159, tag_tid=11, entry_tid=40011], [object_number=158, tag_tid=11, entry_tid=40010], [object_number=157, tag_tid=11, entry_tid=40009], [object_number=156, tag_tid=11, entry_tid=40008]]], state=connected] 2026-02-22T19:58:29.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=159, tag_tid=11, entry_tid=40011' 2026-02-22T19:58:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=159, tag_tid=11, entry_tid=40011' 2026-02-22T19:58:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=159, tag_tid=11, entry_tid=40011' '!=' 'object_number=159, tag_tid=11, entry_tid=39787' 2026-02-22T19:58:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=159, tag_tid=11, entry_tid=40011' = 'object_number=159, tag_tid=11, entry_tid=39787' 2026-02-22T19:58:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=159, tag_tid=11, entry_tid=40011' 2026-02-22T19:58:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=159, tag_tid=11, entry_tid=40011' 2026-02-22T19:58:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:29.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=159, tag_tid=11, entry_tid=40011' 2026-02-22T19:58:29.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:29.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:29.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40011 2026-02-22T19:58:29.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 40011 2026-02-22T19:58:29.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:29.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:29.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:30.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:30.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:30.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:30.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:30.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:30.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:30.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:30.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:30.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:30.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:30.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:30.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:30.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=40224], [object_number=159, tag_tid=11, entry_tid=40223], [object_number=158, tag_tid=11, entry_tid=40222], [object_number=157, tag_tid=11, entry_tid=40221]]], state=connected] 2026-02-22T19:58:30.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:30.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:30.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:30.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:30.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:30.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:30.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=40224], [object_number=159, tag_tid=11, entry_tid=40223], [object_number=158, tag_tid=11, entry_tid=40222], [object_number=157, tag_tid=11, entry_tid=40221]]], state=connected] 2026-02-22T19:58:30.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=156, tag_tid=11, entry_tid=40224' 2026-02-22T19:58:30.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=156, tag_tid=11, entry_tid=40224' 2026-02-22T19:58:30.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=40224' '!=' 'object_number=159, tag_tid=11, entry_tid=40011' 2026-02-22T19:58:30.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:30.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=40224' = 'object_number=159, tag_tid=11, entry_tid=40011' 2026-02-22T19:58:30.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=156, tag_tid=11, entry_tid=40224' 2026-02-22T19:58:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=40224' 2026-02-22T19:58:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:30.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:30.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:30.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:30.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=40224' 2026-02-22T19:58:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:30.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40224 2026-02-22T19:58:30.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 40224 2026-02-22T19:58:30.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:30.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:30.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:30.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:30.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:30.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:30.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:30.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:30.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:30.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:30.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:30.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:30.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:30.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:30.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:30.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:30.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:30.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=40453], [object_number=156, tag_tid=11, entry_tid=40452], [object_number=159, tag_tid=11, entry_tid=40451], [object_number=158, tag_tid=11, entry_tid=40450]]], state=connected] 2026-02-22T19:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:30.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:30.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:30.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:30.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:30.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=40453], [object_number=156, tag_tid=11, entry_tid=40452], [object_number=159, tag_tid=11, entry_tid=40451], [object_number=158, tag_tid=11, entry_tid=40450]]], state=connected] 2026-02-22T19:58:30.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=157, tag_tid=11, entry_tid=40453' 2026-02-22T19:58:30.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=157, tag_tid=11, entry_tid=40453' 2026-02-22T19:58:30.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=40453' '!=' 'object_number=156, tag_tid=11, entry_tid=40224' 2026-02-22T19:58:30.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:30.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=40453' = 'object_number=156, tag_tid=11, entry_tid=40224' 2026-02-22T19:58:30.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=157, tag_tid=11, entry_tid=40453' 2026-02-22T19:58:30.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:30.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:30.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:30.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:30.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=40453' 2026-02-22T19:58:30.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:30.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:30.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:30.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:30.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:30.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:30.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:30.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=40453' 2026-02-22T19:58:30.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:30.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40453 2026-02-22T19:58:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 40453 2026-02-22T19:58:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:30.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:30.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:30.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:30.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:30.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:30.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:30.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:30.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:30.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:30.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:30.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:30.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:30.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:30.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:30.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:30.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:30.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:30.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:30.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:30.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:30.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:30.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:30.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:30.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:30.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:30.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:30.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=40629], [object_number=156, tag_tid=11, entry_tid=40628], [object_number=159, tag_tid=11, entry_tid=40627], [object_number=158, tag_tid=11, entry_tid=40626]]], state=connected] 2026-02-22T19:58:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:30.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:30.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:30.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:30.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:30.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:31.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:31.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:31.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:31.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:31.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=40629], [object_number=156, tag_tid=11, entry_tid=40628], [object_number=159, tag_tid=11, entry_tid=40627], [object_number=158, tag_tid=11, entry_tid=40626]]], state=connected] 2026-02-22T19:58:31.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:31.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=157, tag_tid=11, entry_tid=40629' 2026-02-22T19:58:31.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=157, tag_tid=11, entry_tid=40629' 2026-02-22T19:58:31.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=40629' '!=' 'object_number=157, tag_tid=11, entry_tid=40453' 2026-02-22T19:58:31.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:31.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=40629' = 'object_number=157, tag_tid=11, entry_tid=40453' 2026-02-22T19:58:31.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=157, tag_tid=11, entry_tid=40629' 2026-02-22T19:58:31.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:31.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:31.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:31.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:31.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=40629' 2026-02-22T19:58:31.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:31.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:31.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:31.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:31.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:31.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=40629' 2026-02-22T19:58:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:31.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40629 2026-02-22T19:58:31.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 40629 2026-02-22T19:58:31.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:31.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:31.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:31.155 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T19:58:31.146+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb9d40 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:31.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:31.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:31.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:31.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:31.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:31.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:31.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:31.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:31.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:31.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:31.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:31.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:31.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=159, tag_tid=11, entry_tid=40959], [object_number=158, tag_tid=11, entry_tid=40958], [object_number=157, tag_tid=11, entry_tid=40957], [object_number=156, tag_tid=11, entry_tid=40956]]], state=connected] 2026-02-22T19:58:31.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:31.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:31.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:31.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:31.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:31.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:31.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:31.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:31.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:31.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:31.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:31.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=159, tag_tid=11, entry_tid=40959], [object_number=158, tag_tid=11, entry_tid=40958], [object_number=157, tag_tid=11, entry_tid=40957], [object_number=156, tag_tid=11, entry_tid=40956]]], state=connected] 2026-02-22T19:58:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=159, tag_tid=11, entry_tid=40959' 2026-02-22T19:58:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=159, tag_tid=11, entry_tid=40959' 2026-02-22T19:58:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=159, tag_tid=11, entry_tid=40959' '!=' 'object_number=157, tag_tid=11, entry_tid=40629' 2026-02-22T19:58:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=159, tag_tid=11, entry_tid=40959' = 'object_number=157, tag_tid=11, entry_tid=40629' 2026-02-22T19:58:31.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=159, tag_tid=11, entry_tid=40959' 2026-02-22T19:58:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:31.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=159, tag_tid=11, entry_tid=40959' 2026-02-22T19:58:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:31.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:31.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:31.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=159, tag_tid=11, entry_tid=40959' 2026-02-22T19:58:31.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:31.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40959 2026-02-22T19:58:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 40959 2026-02-22T19:58:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:31.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:31.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:31.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:31.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:31.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:31.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:31.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:31.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:31.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:31.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:31.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:31.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:31.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:31.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:31.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:31.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:31.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:31.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:31.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:31.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:31.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:31.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:32.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:32.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:32.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:32.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=41177], [object_number=156, tag_tid=11, entry_tid=41176], [object_number=159, tag_tid=11, entry_tid=41175], [object_number=158, tag_tid=11, entry_tid=41174]]], state=connected] 2026-02-22T19:58:32.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:32.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:32.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:32.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:32.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:32.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:32.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:32.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:32.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:32.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:32.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:32.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:32.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=41177], [object_number=156, tag_tid=11, entry_tid=41176], [object_number=159, tag_tid=11, entry_tid=41175], [object_number=158, tag_tid=11, entry_tid=41174]]], state=connected] 2026-02-22T19:58:32.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=157, tag_tid=11, entry_tid=41177' 2026-02-22T19:58:32.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=157, tag_tid=11, entry_tid=41177' 2026-02-22T19:58:32.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=41177' '!=' 'object_number=159, tag_tid=11, entry_tid=40959' 2026-02-22T19:58:32.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:32.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=41177' = 'object_number=159, tag_tid=11, entry_tid=40959' 2026-02-22T19:58:32.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=157, tag_tid=11, entry_tid=41177' 2026-02-22T19:58:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:32.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=41177' 2026-02-22T19:58:32.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:32.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:32.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:32.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:32.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:32.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=41177' 2026-02-22T19:58:32.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:32.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41177 2026-02-22T19:58:32.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 41177 2026-02-22T19:58:32.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:32.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:32.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:32.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:32.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:32.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:32.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:32.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:32.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:32.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:32.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:32.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:32.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:32.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:32.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:32.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:32.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:32.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:32.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:32.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=41500], [object_number=159, tag_tid=11, entry_tid=41499], [object_number=158, tag_tid=11, entry_tid=41498], [object_number=157, tag_tid=11, entry_tid=41497]]], state=connected] 2026-02-22T19:58:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:32.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:32.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:32.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:32.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:32.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:32.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:32.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=41500], [object_number=159, tag_tid=11, entry_tid=41499], [object_number=158, tag_tid=11, entry_tid=41498], [object_number=157, tag_tid=11, entry_tid=41497]]], state=connected] 2026-02-22T19:58:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=156, tag_tid=11, entry_tid=41500' 2026-02-22T19:58:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=156, tag_tid=11, entry_tid=41500' 2026-02-22T19:58:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=41500' '!=' 'object_number=157, tag_tid=11, entry_tid=41177' 2026-02-22T19:58:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=41500' = 'object_number=157, tag_tid=11, entry_tid=41177' 2026-02-22T19:58:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=156, tag_tid=11, entry_tid=41500' 2026-02-22T19:58:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:32.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=41500' 2026-02-22T19:58:32.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:32.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:32.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=41500' 2026-02-22T19:58:32.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:32.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41500 2026-02-22T19:58:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 41500 2026-02-22T19:58:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:32.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:32.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:32.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:32.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:32.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:32.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:32.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:32.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:32.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:32.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:32.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:32.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:32.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:32.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:32.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:32.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=158, tag_tid=11, entry_tid=41718], [object_number=157, tag_tid=11, entry_tid=41717], [object_number=156, tag_tid=11, entry_tid=41716], [object_number=159, tag_tid=11, entry_tid=41715]]], state=connected] 2026-02-22T19:58:32.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:32.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:32.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:32.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:32.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:32.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:32.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:32.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:32.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:32.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:32.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:32.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:32.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:32.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:32.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:32.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=158, tag_tid=11, entry_tid=41718], [object_number=157, tag_tid=11, entry_tid=41717], [object_number=156, tag_tid=11, entry_tid=41716], [object_number=159, tag_tid=11, entry_tid=41715]]], state=connected] 2026-02-22T19:58:32.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:32.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=158, tag_tid=11, entry_tid=41718' 2026-02-22T19:58:32.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=158, tag_tid=11, entry_tid=41718' 2026-02-22T19:58:32.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=158, tag_tid=11, entry_tid=41718' '!=' 'object_number=156, tag_tid=11, entry_tid=41500' 2026-02-22T19:58:32.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:32.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=158, tag_tid=11, entry_tid=41718' = 'object_number=156, tag_tid=11, entry_tid=41500' 2026-02-22T19:58:32.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=158, tag_tid=11, entry_tid=41718' 2026-02-22T19:58:32.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:32.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:32.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:32.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=158, tag_tid=11, entry_tid=41718' 2026-02-22T19:58:32.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:32.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:32.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:32.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:32.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:32.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=158, tag_tid=11, entry_tid=41718' 2026-02-22T19:58:32.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:32.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:32.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41718 2026-02-22T19:58:32.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 41718 2026-02-22T19:58:32.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:32.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:32.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:33.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:33.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:33.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:33.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:33.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:33.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:33.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:33.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:33.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:33.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:33.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:33.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:33.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:33.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:33.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:33.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:33.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:33.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:33.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:33.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:33.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:33.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=41912], [object_number=159, tag_tid=11, entry_tid=41911], [object_number=158, tag_tid=11, entry_tid=41910], [object_number=157, tag_tid=11, entry_tid=41909]]], state=connected] 2026-02-22T19:58:33.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:33.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:33.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:33.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:33.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=41912], [object_number=159, tag_tid=11, entry_tid=41911], [object_number=158, tag_tid=11, entry_tid=41910], [object_number=157, tag_tid=11, entry_tid=41909]]], state=connected] 2026-02-22T19:58:33.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=156, tag_tid=11, entry_tid=41912' 2026-02-22T19:58:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=156, tag_tid=11, entry_tid=41912' 2026-02-22T19:58:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=41912' '!=' 'object_number=158, tag_tid=11, entry_tid=41718' 2026-02-22T19:58:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=41912' = 'object_number=158, tag_tid=11, entry_tid=41718' 2026-02-22T19:58:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=156, tag_tid=11, entry_tid=41912' 2026-02-22T19:58:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=41912' 2026-02-22T19:58:33.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:33.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:33.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=41912' 2026-02-22T19:58:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41912 2026-02-22T19:58:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 41912 2026-02-22T19:58:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:33.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:33.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:33.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:33.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:33.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:33.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:33.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:33.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:33.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:33.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:33.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:33.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:33.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:33.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:33.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:33.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:33.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:33.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:33.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:33.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:33.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:33.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:33.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:33.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:33.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:33.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:33.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:33.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:33.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:33.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:33.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:33.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:33.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:33.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:33.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=42109], [object_number=156, tag_tid=11, entry_tid=42108], [object_number=159, tag_tid=11, entry_tid=42107], [object_number=158, tag_tid=11, entry_tid=42106]]], state=connected] 2026-02-22T19:58:33.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:33.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:33.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:33.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:33.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:33.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:33.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:33.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:33.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:33.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=42109], [object_number=156, tag_tid=11, entry_tid=42108], [object_number=159, tag_tid=11, entry_tid=42107], [object_number=158, tag_tid=11, entry_tid=42106]]], state=connected] 2026-02-22T19:58:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=157, tag_tid=11, entry_tid=42109' 2026-02-22T19:58:33.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=157, tag_tid=11, entry_tid=42109' 2026-02-22T19:58:33.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=42109' '!=' 'object_number=156, tag_tid=11, entry_tid=41912' 2026-02-22T19:58:33.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:33.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=42109' = 'object_number=156, tag_tid=11, entry_tid=41912' 2026-02-22T19:58:33.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=157, tag_tid=11, entry_tid=42109' 2026-02-22T19:58:33.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=42109' 2026-02-22T19:58:33.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:33.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:33.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:33.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:33.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:33.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=42109' 2026-02-22T19:58:33.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:33.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42109 2026-02-22T19:58:33.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 42109 2026-02-22T19:58:33.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:33.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:33.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:33.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:33.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:33.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:33.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:33.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:33.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:33.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:33.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:33.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=42325], [object_number=156, tag_tid=11, entry_tid=42324], [object_number=159, tag_tid=11, entry_tid=42323], [object_number=158, tag_tid=11, entry_tid=42322]]], state=connected] 2026-02-22T19:58:33.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:33.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:33.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:33.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:33.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:33.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:33.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:33.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:33.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=157, tag_tid=11, entry_tid=42325], [object_number=156, tag_tid=11, entry_tid=42324], [object_number=159, tag_tid=11, entry_tid=42323], [object_number=158, tag_tid=11, entry_tid=42322]]], state=connected] 2026-02-22T19:58:33.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=157, tag_tid=11, entry_tid=42325' 2026-02-22T19:58:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=157, tag_tid=11, entry_tid=42325' 2026-02-22T19:58:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=42325' '!=' 'object_number=157, tag_tid=11, entry_tid=42109' 2026-02-22T19:58:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=157, tag_tid=11, entry_tid=42325' = 'object_number=157, tag_tid=11, entry_tid=42109' 2026-02-22T19:58:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=157, tag_tid=11, entry_tid=42325' 2026-02-22T19:58:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:33.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=42325' 2026-02-22T19:58:33.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:33.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:33.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:33.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:33.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:33.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=157, tag_tid=11, entry_tid=42325' 2026-02-22T19:58:33.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:33.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:33.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42325 2026-02-22T19:58:33.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 42325 2026-02-22T19:58:33.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:33.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:33.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:34.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:34.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:34.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:34.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:34.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:34.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:34.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:34.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:34.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:34.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:34.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:34.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:34.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:34.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:34.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:34.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:34.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:34.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:34.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:34.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:34.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:34.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:34.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:34.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:34.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:34.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:34.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=42572], [object_number=159, tag_tid=11, entry_tid=42571], [object_number=158, tag_tid=11, entry_tid=42570], [object_number=157, tag_tid=11, entry_tid=42569]]], state=connected] 2026-02-22T19:58:34.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:34.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:34.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:34.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:34.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:34.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:34.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:34.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:34.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:34.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=42572], [object_number=159, tag_tid=11, entry_tid=42571], [object_number=158, tag_tid=11, entry_tid=42570], [object_number=157, tag_tid=11, entry_tid=42569]]], state=connected] 2026-02-22T19:58:34.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=156, tag_tid=11, entry_tid=42572' 2026-02-22T19:58:34.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=156, tag_tid=11, entry_tid=42572' 2026-02-22T19:58:34.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=42572' '!=' 'object_number=157, tag_tid=11, entry_tid=42325' 2026-02-22T19:58:34.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:34.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=42572' = 'object_number=157, tag_tid=11, entry_tid=42325' 2026-02-22T19:58:34.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=156, tag_tid=11, entry_tid=42572' 2026-02-22T19:58:34.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:34.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:34.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:34.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=42572' 2026-02-22T19:58:34.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:34.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:34.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:34.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:34.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=42572' 2026-02-22T19:58:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42572 2026-02-22T19:58:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 42572 2026-02-22T19:58:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:34.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:34.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:34.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:34.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:34.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:34.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:34.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:34.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:34.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:34.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:34.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:34.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:34.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:34.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:34.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:34.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:34.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:34.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:34.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:34.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:34.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:34.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:34.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:34.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:34.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:34.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:34.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:34.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:34.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:34.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:34.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:34.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:34.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:34.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:34.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:34.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:34.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:34.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:34.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:34.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:34.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:34.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:34.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:34.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:34.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:34.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:34.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:34.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:34.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:34.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:34.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:34.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:34.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:34.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:34.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=42804], [object_number=159, tag_tid=11, entry_tid=42803], [object_number=158, tag_tid=11, entry_tid=42802], [object_number=157, tag_tid=11, entry_tid=42801]]], state=connected] 2026-02-22T19:58:34.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:34.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:34.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:34.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:34.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:34.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=42804], [object_number=159, tag_tid=11, entry_tid=42803], [object_number=158, tag_tid=11, entry_tid=42802], [object_number=157, tag_tid=11, entry_tid=42801]]], state=connected] 2026-02-22T19:58:34.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=156, tag_tid=11, entry_tid=42804' 2026-02-22T19:58:34.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=156, tag_tid=11, entry_tid=42804' 2026-02-22T19:58:34.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=42804' '!=' 'object_number=156, tag_tid=11, entry_tid=42572' 2026-02-22T19:58:34.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:34.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=42804' = 'object_number=156, tag_tid=11, entry_tid=42572' 2026-02-22T19:58:34.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=156, tag_tid=11, entry_tid=42804' 2026-02-22T19:58:34.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:34.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:34.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:34.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=42804' 2026-02-22T19:58:34.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:34.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:34.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:34.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:34.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:34.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=42804' 2026-02-22T19:58:34.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:34.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42804 2026-02-22T19:58:34.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 42804 2026-02-22T19:58:34.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:34.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:34.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:34.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:34.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:34.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:34.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:34.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:34.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:34.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:34.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:34.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:34.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:34.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:34.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:34.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:34.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:34.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=159, tag_tid=11, entry_tid=43019], [object_number=158, tag_tid=11, entry_tid=43018], [object_number=157, tag_tid=11, entry_tid=43017], [object_number=156, tag_tid=11, entry_tid=43016]]], state=connected] 2026-02-22T19:58:34.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:34.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:34.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:34.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:34.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:34.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:34.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:34.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:34.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:34.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=159, tag_tid=11, entry_tid=43019], [object_number=158, tag_tid=11, entry_tid=43018], [object_number=157, tag_tid=11, entry_tid=43017], [object_number=156, tag_tid=11, entry_tid=43016]]], state=connected] 2026-02-22T19:58:34.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=159, tag_tid=11, entry_tid=43019' 2026-02-22T19:58:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=159, tag_tid=11, entry_tid=43019' 2026-02-22T19:58:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=159, tag_tid=11, entry_tid=43019' '!=' 'object_number=156, tag_tid=11, entry_tid=42804' 2026-02-22T19:58:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=159, tag_tid=11, entry_tid=43019' = 'object_number=156, tag_tid=11, entry_tid=42804' 2026-02-22T19:58:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=159, tag_tid=11, entry_tid=43019' 2026-02-22T19:58:34.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:34.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:34.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:34.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=159, tag_tid=11, entry_tid=43019' 2026-02-22T19:58:34.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:34.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=159, tag_tid=11, entry_tid=43019' 2026-02-22T19:58:34.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:34.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43019 2026-02-22T19:58:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 43019 2026-02-22T19:58:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:34.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:35.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:35.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:35.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:35.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:35.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:35.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:35.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:35.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:35.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:35.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:35.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:35.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:35.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=43236], [object_number=159, tag_tid=11, entry_tid=43235], [object_number=158, tag_tid=11, entry_tid=43234], [object_number=157, tag_tid=11, entry_tid=43233]]], state=connected] 2026-02-22T19:58:35.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:35.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:35.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=43236], [object_number=159, tag_tid=11, entry_tid=43235], [object_number=158, tag_tid=11, entry_tid=43234], [object_number=157, tag_tid=11, entry_tid=43233]]], state=connected] 2026-02-22T19:58:35.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=156, tag_tid=11, entry_tid=43236' 2026-02-22T19:58:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=156, tag_tid=11, entry_tid=43236' 2026-02-22T19:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=43236' '!=' 'object_number=159, tag_tid=11, entry_tid=43019' 2026-02-22T19:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=43236' = 'object_number=159, tag_tid=11, entry_tid=43019' 2026-02-22T19:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=156, tag_tid=11, entry_tid=43236' 2026-02-22T19:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:35.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:35.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:35.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=43236' 2026-02-22T19:58:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:35.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:35.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:35.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:35.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:35.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=43236' 2026-02-22T19:58:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:35.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43236 2026-02-22T19:58:35.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 43236 2026-02-22T19:58:35.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:35.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:35.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:35.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:35.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:35.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:35.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:35.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:35.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:35.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:35.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:35.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:35.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:35.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:35.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:35.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:35.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:35.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:35.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:35.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:35.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:35.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:35.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:35.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=43472], [object_number=159, tag_tid=11, entry_tid=43471], [object_number=158, tag_tid=11, entry_tid=43470], [object_number=157, tag_tid=11, entry_tid=43469]]], state=connected] 2026-02-22T19:58:35.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:35.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:35.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:35.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=43472], [object_number=159, tag_tid=11, entry_tid=43471], [object_number=158, tag_tid=11, entry_tid=43470], [object_number=157, tag_tid=11, entry_tid=43469]]], state=connected] 2026-02-22T19:58:35.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:35.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=156, tag_tid=11, entry_tid=43472' 2026-02-22T19:58:35.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=156, tag_tid=11, entry_tid=43472' 2026-02-22T19:58:35.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=43472' '!=' 'object_number=156, tag_tid=11, entry_tid=43236' 2026-02-22T19:58:35.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:35.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=43472' = 'object_number=156, tag_tid=11, entry_tid=43236' 2026-02-22T19:58:35.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=156, tag_tid=11, entry_tid=43472' 2026-02-22T19:58:35.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:35.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:35.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:35.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:35.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=43472' 2026-02-22T19:58:35.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:35.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:35.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:35.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:35.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:35.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:35.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:35.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=43472' 2026-02-22T19:58:35.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:35.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:35.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43472 2026-02-22T19:58:35.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 43472 2026-02-22T19:58:35.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:35.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:35.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:35.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:35.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:35.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:35.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:35.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:35.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:35.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:35.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:35.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:35.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:35.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:35.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:35.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:35.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:36.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:36.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:36.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:36.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:36.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=158, tag_tid=11, entry_tid=43690], [object_number=157, tag_tid=11, entry_tid=43689], [object_number=156, tag_tid=11, entry_tid=43688], [object_number=159, tag_tid=11, entry_tid=43687]]], state=connected] 2026-02-22T19:58:36.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:36.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:36.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=158, tag_tid=11, entry_tid=43690], [object_number=157, tag_tid=11, entry_tid=43689], [object_number=156, tag_tid=11, entry_tid=43688], [object_number=159, tag_tid=11, entry_tid=43687]]], state=connected] 2026-02-22T19:58:36.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=158, tag_tid=11, entry_tid=43690' 2026-02-22T19:58:36.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=158, tag_tid=11, entry_tid=43690' 2026-02-22T19:58:36.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=158, tag_tid=11, entry_tid=43690' '!=' 'object_number=156, tag_tid=11, entry_tid=43472' 2026-02-22T19:58:36.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:36.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=158, tag_tid=11, entry_tid=43690' = 'object_number=156, tag_tid=11, entry_tid=43472' 2026-02-22T19:58:36.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=158, tag_tid=11, entry_tid=43690' 2026-02-22T19:58:36.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:36.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:36.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=158, tag_tid=11, entry_tid=43690' 2026-02-22T19:58:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=158, tag_tid=11, entry_tid=43690' 2026-02-22T19:58:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43690 2026-02-22T19:58:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 43690 2026-02-22T19:58:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:36.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:36.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:36.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:36.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:36.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:36.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:36.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:36.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:36.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:36.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=43888], [object_number=159, tag_tid=11, entry_tid=43887], [object_number=158, tag_tid=11, entry_tid=43886], [object_number=157, tag_tid=11, entry_tid=43885]]], state=connected] 2026-02-22T19:58:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:36.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:36.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:36.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:36.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 39 2026-02-22T19:58:36.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:36.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:36.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:36.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=156, tag_tid=11, entry_tid=43888], [object_number=159, tag_tid=11, entry_tid=43887], [object_number=158, tag_tid=11, entry_tid=43886], [object_number=157, tag_tid=11, entry_tid=43885]]], state=connected] 2026-02-22T19:58:36.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=156, tag_tid=11, entry_tid=43888' 2026-02-22T19:58:36.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=156, tag_tid=11, entry_tid=43888' 2026-02-22T19:58:36.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=43888' '!=' 'object_number=158, tag_tid=11, entry_tid=43690' 2026-02-22T19:58:36.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:36.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=156, tag_tid=11, entry_tid=43888' = 'object_number=158, tag_tid=11, entry_tid=43690' 2026-02-22T19:58:36.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=156, tag_tid=11, entry_tid=43888' 2026-02-22T19:58:36.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:36.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:36.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:36.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=43888' 2026-02-22T19:58:36.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:36.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:36.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:36.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:36.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:36.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=156, tag_tid=11, entry_tid=43888' 2026-02-22T19:58:36.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:36.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43888 2026-02-22T19:58:36.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 43888 2026-02-22T19:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:36.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:36.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:36.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:36.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:36.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:36.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:36.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:36.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:36.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:36.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:36.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:36.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=44035], [object_number=162, tag_tid=11, entry_tid=44034], [object_number=161, tag_tid=11, entry_tid=44033], [object_number=160, tag_tid=11, entry_tid=44032]]], state=connected] 2026-02-22T19:58:36.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:36.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:36.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:36.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:36.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:36.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:36.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=44035], [object_number=162, tag_tid=11, entry_tid=44034], [object_number=161, tag_tid=11, entry_tid=44033], [object_number=160, tag_tid=11, entry_tid=44032]]], state=connected] 2026-02-22T19:58:36.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:36.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=163, tag_tid=11, entry_tid=44035' 2026-02-22T19:58:36.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=163, tag_tid=11, entry_tid=44035' 2026-02-22T19:58:36.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=44035' '!=' 'object_number=156, tag_tid=11, entry_tid=43888' 2026-02-22T19:58:36.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:36.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=44035' = 'object_number=156, tag_tid=11, entry_tid=43888' 2026-02-22T19:58:36.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=163, tag_tid=11, entry_tid=44035' 2026-02-22T19:58:36.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:36.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:36.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:36.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=44035' 2026-02-22T19:58:36.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:36.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:36.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:36.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:36.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:36.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=44035' 2026-02-22T19:58:36.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:36.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:36.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44035 2026-02-22T19:58:36.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 44035 2026-02-22T19:58:36.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:36.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:36.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:37.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:37.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:37.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:37.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:37.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:37.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:37.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:37.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:37.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:37.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:37.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:37.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:37.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:37.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:37.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:37.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:37.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:37.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:37.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:37.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:37.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:37.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:37.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:37.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:37.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:37.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:37.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:37.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:37.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:37.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:37.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:37.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=162, tag_tid=11, entry_tid=44226], [object_number=161, tag_tid=11, entry_tid=44225], [object_number=160, tag_tid=11, entry_tid=44224], [object_number=163, tag_tid=11, entry_tid=44223]]], state=connected] 2026-02-22T19:58:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:37.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:37.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:37.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:37.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:37.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:37.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:37.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:37.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=162, tag_tid=11, entry_tid=44226], [object_number=161, tag_tid=11, entry_tid=44225], [object_number=160, tag_tid=11, entry_tid=44224], [object_number=163, tag_tid=11, entry_tid=44223]]], state=connected] 2026-02-22T19:58:37.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=162, tag_tid=11, entry_tid=44226' 2026-02-22T19:58:37.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=162, tag_tid=11, entry_tid=44226' 2026-02-22T19:58:37.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=162, tag_tid=11, entry_tid=44226' '!=' 'object_number=163, tag_tid=11, entry_tid=44035' 2026-02-22T19:58:37.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:37.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=162, tag_tid=11, entry_tid=44226' = 'object_number=163, tag_tid=11, entry_tid=44035' 2026-02-22T19:58:37.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=162, tag_tid=11, entry_tid=44226' 2026-02-22T19:58:37.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:37.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:37.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:37.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=162, tag_tid=11, entry_tid=44226' 2026-02-22T19:58:37.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:37.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:37.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:37.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:37.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:37.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=162, tag_tid=11, entry_tid=44226' 2026-02-22T19:58:37.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:37.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44226 2026-02-22T19:58:37.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 44226 2026-02-22T19:58:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:37.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:37.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:37.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:37.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:37.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:37.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:37.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:37.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:37.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:37.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:37.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:37.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:37.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:37.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:37.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:37.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:37.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:37.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:37.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:37.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=44437], [object_number=160, tag_tid=11, entry_tid=44436], [object_number=163, tag_tid=11, entry_tid=44435], [object_number=162, tag_tid=11, entry_tid=44434]]], state=connected] 2026-02-22T19:58:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:37.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:37.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:37.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:37.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:37.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:37.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=44437], [object_number=160, tag_tid=11, entry_tid=44436], [object_number=163, tag_tid=11, entry_tid=44435], [object_number=162, tag_tid=11, entry_tid=44434]]], state=connected] 2026-02-22T19:58:37.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=161, tag_tid=11, entry_tid=44437' 2026-02-22T19:58:37.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=161, tag_tid=11, entry_tid=44437' 2026-02-22T19:58:37.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=44437' '!=' 'object_number=162, tag_tid=11, entry_tid=44226' 2026-02-22T19:58:37.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:37.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=44437' = 'object_number=162, tag_tid=11, entry_tid=44226' 2026-02-22T19:58:37.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=161, tag_tid=11, entry_tid=44437' 2026-02-22T19:58:37.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:37.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:37.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=44437' 2026-02-22T19:58:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:37.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:37.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:37.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:37.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=44437' 2026-02-22T19:58:37.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:37.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44437 2026-02-22T19:58:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 44437 2026-02-22T19:58:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:37.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:37.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:37.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:37.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:37.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:37.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:37.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:37.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:37.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:37.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:37.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:37.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=162, tag_tid=11, entry_tid=44630], [object_number=161, tag_tid=11, entry_tid=44629], [object_number=160, tag_tid=11, entry_tid=44628], [object_number=163, tag_tid=11, entry_tid=44627]]], state=connected] 2026-02-22T19:58:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:37.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:37.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:37.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:37.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:37.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:37.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:37.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:37.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=162, tag_tid=11, entry_tid=44630], [object_number=161, tag_tid=11, entry_tid=44629], [object_number=160, tag_tid=11, entry_tid=44628], [object_number=163, tag_tid=11, entry_tid=44627]]], state=connected] 2026-02-22T19:58:37.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=162, tag_tid=11, entry_tid=44630' 2026-02-22T19:58:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=162, tag_tid=11, entry_tid=44630' 2026-02-22T19:58:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=162, tag_tid=11, entry_tid=44630' '!=' 'object_number=161, tag_tid=11, entry_tid=44437' 2026-02-22T19:58:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=162, tag_tid=11, entry_tid=44630' = 'object_number=161, tag_tid=11, entry_tid=44437' 2026-02-22T19:58:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=162, tag_tid=11, entry_tid=44630' 2026-02-22T19:58:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=162, tag_tid=11, entry_tid=44630' 2026-02-22T19:58:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:37.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:37.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:37.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=162, tag_tid=11, entry_tid=44630' 2026-02-22T19:58:37.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:37.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44630 2026-02-22T19:58:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 44630 2026-02-22T19:58:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:38.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:38.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:38.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:38.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:38.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:38.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:38.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:38.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:38.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:38.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:38.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:38.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:38.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:38.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:38.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:38.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=44835], [object_number=162, tag_tid=11, entry_tid=44834], [object_number=161, tag_tid=11, entry_tid=44833], [object_number=160, tag_tid=11, entry_tid=44832]]], state=connected] 2026-02-22T19:58:38.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:38.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:38.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:38.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:38.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:38.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=44835], [object_number=162, tag_tid=11, entry_tid=44834], [object_number=161, tag_tid=11, entry_tid=44833], [object_number=160, tag_tid=11, entry_tid=44832]]], state=connected] 2026-02-22T19:58:38.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=163, tag_tid=11, entry_tid=44835' 2026-02-22T19:58:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=163, tag_tid=11, entry_tid=44835' 2026-02-22T19:58:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=44835' '!=' 'object_number=162, tag_tid=11, entry_tid=44630' 2026-02-22T19:58:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=44835' = 'object_number=162, tag_tid=11, entry_tid=44630' 2026-02-22T19:58:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=163, tag_tid=11, entry_tid=44835' 2026-02-22T19:58:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:38.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:38.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:38.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=44835' 2026-02-22T19:58:38.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:38.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:38.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:38.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:38.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:38.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:38.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:38.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=44835' 2026-02-22T19:58:38.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:38.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:38.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44835 2026-02-22T19:58:38.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 44835 2026-02-22T19:58:38.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:38.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:38.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:38.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:38.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:38.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:38.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:38.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:38.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:38.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:38.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:38.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:38.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:38.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:38.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:38.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:38.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:38.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:38.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:38.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=45047], [object_number=162, tag_tid=11, entry_tid=45046], [object_number=161, tag_tid=11, entry_tid=45045], [object_number=160, tag_tid=11, entry_tid=45044]]], state=connected] 2026-02-22T19:58:38.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:38.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:38.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:38.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:38.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:38.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:38.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:38.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:38.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=45047], [object_number=162, tag_tid=11, entry_tid=45046], [object_number=161, tag_tid=11, entry_tid=45045], [object_number=160, tag_tid=11, entry_tid=45044]]], state=connected] 2026-02-22T19:58:38.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=163, tag_tid=11, entry_tid=45047' 2026-02-22T19:58:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=163, tag_tid=11, entry_tid=45047' 2026-02-22T19:58:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=45047' '!=' 'object_number=163, tag_tid=11, entry_tid=44835' 2026-02-22T19:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=45047' = 'object_number=163, tag_tid=11, entry_tid=44835' 2026-02-22T19:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=163, tag_tid=11, entry_tid=45047' 2026-02-22T19:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=45047' 2026-02-22T19:58:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:38.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:38.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:38.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:38.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=45047' 2026-02-22T19:58:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45047 2026-02-22T19:58:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 45047 2026-02-22T19:58:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:38.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:38.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:38.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:38.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:38.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:38.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:38.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:38.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:38.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:38.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:38.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:38.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:38.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=45249], [object_number=160, tag_tid=11, entry_tid=45248], [object_number=163, tag_tid=11, entry_tid=45247], [object_number=162, tag_tid=11, entry_tid=45246]]], state=connected] 2026-02-22T19:58:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:38.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:38.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:38.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:38.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:38.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:38.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:39.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=45249], [object_number=160, tag_tid=11, entry_tid=45248], [object_number=163, tag_tid=11, entry_tid=45247], [object_number=162, tag_tid=11, entry_tid=45246]]], state=connected] 2026-02-22T19:58:39.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=161, tag_tid=11, entry_tid=45249' 2026-02-22T19:58:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=161, tag_tid=11, entry_tid=45249' 2026-02-22T19:58:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=45249' '!=' 'object_number=163, tag_tid=11, entry_tid=45047' 2026-02-22T19:58:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=45249' = 'object_number=163, tag_tid=11, entry_tid=45047' 2026-02-22T19:58:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=161, tag_tid=11, entry_tid=45249' 2026-02-22T19:58:39.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:39.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:39.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:39.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=45249' 2026-02-22T19:58:39.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:39.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:39.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:39.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:39.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:39.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=45249' 2026-02-22T19:58:39.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:39.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45249 2026-02-22T19:58:39.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 45249 2026-02-22T19:58:39.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:39.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:39.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:39.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:39.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:39.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:39.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:39.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:39.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:39.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:39.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:39.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:39.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:39.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:39.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:39.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:39.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:39.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:39.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:39.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:39.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:39.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:39.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=45465], [object_number=160, tag_tid=11, entry_tid=45464], [object_number=163, tag_tid=11, entry_tid=45463], [object_number=162, tag_tid=11, entry_tid=45462]]], state=connected] 2026-02-22T19:58:39.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:39.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:39.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:39.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:39.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:39.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:39.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:39.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:39.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:39.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:39.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:39.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:39.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:39.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:39.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:39.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:39.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:39.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:39.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=45465], [object_number=160, tag_tid=11, entry_tid=45464], [object_number=163, tag_tid=11, entry_tid=45463], [object_number=162, tag_tid=11, entry_tid=45462]]], state=connected] 2026-02-22T19:58:39.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=161, tag_tid=11, entry_tid=45465' 2026-02-22T19:58:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=161, tag_tid=11, entry_tid=45465' 2026-02-22T19:58:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=45465' '!=' 'object_number=161, tag_tid=11, entry_tid=45249' 2026-02-22T19:58:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=45465' = 'object_number=161, tag_tid=11, entry_tid=45249' 2026-02-22T19:58:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=161, tag_tid=11, entry_tid=45465' 2026-02-22T19:58:39.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:39.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:39.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=45465' 2026-02-22T19:58:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:39.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:39.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:39.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:39.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=45465' 2026-02-22T19:58:39.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:39.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45465 2026-02-22T19:58:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 45465 2026-02-22T19:58:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:39.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:39.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:39.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:39.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:39.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:39.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:39.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:39.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:39.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:39.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:39.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:39.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:39.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=162, tag_tid=11, entry_tid=45658], [object_number=161, tag_tid=11, entry_tid=45657], [object_number=160, tag_tid=11, entry_tid=45656], [object_number=163, tag_tid=11, entry_tid=45655]]], state=connected] 2026-02-22T19:58:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:39.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:39.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=162, tag_tid=11, entry_tid=45658], [object_number=161, tag_tid=11, entry_tid=45657], [object_number=160, tag_tid=11, entry_tid=45656], [object_number=163, tag_tid=11, entry_tid=45655]]], state=connected] 2026-02-22T19:58:39.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=162, tag_tid=11, entry_tid=45658' 2026-02-22T19:58:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=162, tag_tid=11, entry_tid=45658' 2026-02-22T19:58:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=162, tag_tid=11, entry_tid=45658' '!=' 'object_number=161, tag_tid=11, entry_tid=45465' 2026-02-22T19:58:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=162, tag_tid=11, entry_tid=45658' = 'object_number=161, tag_tid=11, entry_tid=45465' 2026-02-22T19:58:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=162, tag_tid=11, entry_tid=45658' 2026-02-22T19:58:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:39.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=162, tag_tid=11, entry_tid=45658' 2026-02-22T19:58:39.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:39.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:39.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:39.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:39.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:39.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=162, tag_tid=11, entry_tid=45658' 2026-02-22T19:58:39.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:39.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:39.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45658 2026-02-22T19:58:39.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 45658 2026-02-22T19:58:39.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:39.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:39.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:39.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:39.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:39.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:39.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:39.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:39.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:39.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:39.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:40.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:40.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:40.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:40.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:40.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:40.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:40.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:40.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:40.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=162, tag_tid=11, entry_tid=45850], [object_number=161, tag_tid=11, entry_tid=45849], [object_number=160, tag_tid=11, entry_tid=45848], [object_number=163, tag_tid=11, entry_tid=45847]]], state=connected] 2026-02-22T19:58:40.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:40.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:40.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:40.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:40.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:40.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=162, tag_tid=11, entry_tid=45850], [object_number=161, tag_tid=11, entry_tid=45849], [object_number=160, tag_tid=11, entry_tid=45848], [object_number=163, tag_tid=11, entry_tid=45847]]], state=connected] 2026-02-22T19:58:40.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=162, tag_tid=11, entry_tid=45850' 2026-02-22T19:58:40.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=162, tag_tid=11, entry_tid=45850' 2026-02-22T19:58:40.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=162, tag_tid=11, entry_tid=45850' '!=' 'object_number=162, tag_tid=11, entry_tid=45658' 2026-02-22T19:58:40.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:40.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=162, tag_tid=11, entry_tid=45850' = 'object_number=162, tag_tid=11, entry_tid=45658' 2026-02-22T19:58:40.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=162, tag_tid=11, entry_tid=45850' 2026-02-22T19:58:40.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:40.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:40.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=162, tag_tid=11, entry_tid=45850' 2026-02-22T19:58:40.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:40.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:40.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:40.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:40.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:40.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:40.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:40.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=162, tag_tid=11, entry_tid=45850' 2026-02-22T19:58:40.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:40.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:40.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45850 2026-02-22T19:58:40.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 45850 2026-02-22T19:58:40.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:40.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:40.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:40.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:40.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:40.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:40.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:40.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:40.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:40.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:40.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:40.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:40.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:40.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:40.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:40.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:40.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:40.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:40.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:40.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:40.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:40.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:40.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:40.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:40.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:40.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=46063], [object_number=162, tag_tid=11, entry_tid=46062], [object_number=161, tag_tid=11, entry_tid=46061], [object_number=160, tag_tid=11, entry_tid=46060]]], state=connected] 2026-02-22T19:58:40.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:40.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=46063], [object_number=162, tag_tid=11, entry_tid=46062], [object_number=161, tag_tid=11, entry_tid=46061], [object_number=160, tag_tid=11, entry_tid=46060]]], state=connected] 2026-02-22T19:58:40.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=163, tag_tid=11, entry_tid=46063' 2026-02-22T19:58:40.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=163, tag_tid=11, entry_tid=46063' 2026-02-22T19:58:40.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=46063' '!=' 'object_number=162, tag_tid=11, entry_tid=45850' 2026-02-22T19:58:40.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:40.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=46063' = 'object_number=162, tag_tid=11, entry_tid=45850' 2026-02-22T19:58:40.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=163, tag_tid=11, entry_tid=46063' 2026-02-22T19:58:40.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:40.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:40.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=46063' 2026-02-22T19:58:40.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:40.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:40.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:40.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:40.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:40.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:40.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:40.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=46063' 2026-02-22T19:58:40.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:40.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46063 2026-02-22T19:58:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 46063 2026-02-22T19:58:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:40.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:40.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:40.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:40.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:40.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:40.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:40.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:40.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:40.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:40.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:40.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:40.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:40.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:40.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:40.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:40.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:40.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:40.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:40.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:40.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:40.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:40.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:41.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:41.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:41.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=160, tag_tid=11, entry_tid=46276], [object_number=163, tag_tid=11, entry_tid=46275], [object_number=162, tag_tid=11, entry_tid=46274], [object_number=161, tag_tid=11, entry_tid=46273]]], state=connected] 2026-02-22T19:58:41.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:41.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:41.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:41.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:41.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:41.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:41.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:41.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:41.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=160, tag_tid=11, entry_tid=46276], [object_number=163, tag_tid=11, entry_tid=46275], [object_number=162, tag_tid=11, entry_tid=46274], [object_number=161, tag_tid=11, entry_tid=46273]]], state=connected] 2026-02-22T19:58:41.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=160, tag_tid=11, entry_tid=46276' 2026-02-22T19:58:41.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=160, tag_tid=11, entry_tid=46276' 2026-02-22T19:58:41.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=160, tag_tid=11, entry_tid=46276' '!=' 'object_number=163, tag_tid=11, entry_tid=46063' 2026-02-22T19:58:41.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:41.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=160, tag_tid=11, entry_tid=46276' = 'object_number=163, tag_tid=11, entry_tid=46063' 2026-02-22T19:58:41.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=160, tag_tid=11, entry_tid=46276' 2026-02-22T19:58:41.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:41.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:41.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:41.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=160, tag_tid=11, entry_tid=46276' 2026-02-22T19:58:41.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:41.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:41.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=160, tag_tid=11, entry_tid=46276' 2026-02-22T19:58:41.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:41.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46276 2026-02-22T19:58:41.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 46276 2026-02-22T19:58:41.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:41.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:41.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:41.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:41.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:41.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:41.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:41.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:41.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:41.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:41.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:41.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:41.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:41.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:41.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:41.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:41.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:41.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:41.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:41.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:41.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:41.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:41.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:41.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:41.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:41.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:41.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:41.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:41.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:41.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:41.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=46605], [object_number=160, tag_tid=11, entry_tid=46604], [object_number=163, tag_tid=11, entry_tid=46603], [object_number=162, tag_tid=11, entry_tid=46602]]], state=connected] 2026-02-22T19:58:41.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:41.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:41.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:41.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:41.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:41.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:41.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:41.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:41.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:41.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:41.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:41.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:41.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=46605], [object_number=160, tag_tid=11, entry_tid=46604], [object_number=163, tag_tid=11, entry_tid=46603], [object_number=162, tag_tid=11, entry_tid=46602]]], state=connected] 2026-02-22T19:58:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=161, tag_tid=11, entry_tid=46605' 2026-02-22T19:58:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=161, tag_tid=11, entry_tid=46605' 2026-02-22T19:58:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=46605' '!=' 'object_number=160, tag_tid=11, entry_tid=46276' 2026-02-22T19:58:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=46605' = 'object_number=160, tag_tid=11, entry_tid=46276' 2026-02-22T19:58:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=161, tag_tid=11, entry_tid=46605' 2026-02-22T19:58:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:41.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=46605' 2026-02-22T19:58:41.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:41.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:41.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:41.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:41.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:41.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=46605' 2026-02-22T19:58:41.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:41.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46605 2026-02-22T19:58:41.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 46605 2026-02-22T19:58:41.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:41.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:41.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:41.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:41.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:41.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:41.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:41.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:41.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:41.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:41.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=46837], [object_number=160, tag_tid=11, entry_tid=46836], [object_number=163, tag_tid=11, entry_tid=46835], [object_number=162, tag_tid=11, entry_tid=46834]]], state=connected] 2026-02-22T19:58:41.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:41.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:41.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:41.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:41.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:41.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=46837], [object_number=160, tag_tid=11, entry_tid=46836], [object_number=163, tag_tid=11, entry_tid=46835], [object_number=162, tag_tid=11, entry_tid=46834]]], state=connected] 2026-02-22T19:58:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=161, tag_tid=11, entry_tid=46837' 2026-02-22T19:58:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=161, tag_tid=11, entry_tid=46837' 2026-02-22T19:58:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=46837' '!=' 'object_number=161, tag_tid=11, entry_tid=46605' 2026-02-22T19:58:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=46837' = 'object_number=161, tag_tid=11, entry_tid=46605' 2026-02-22T19:58:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=161, tag_tid=11, entry_tid=46837' 2026-02-22T19:58:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:41.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=46837' 2026-02-22T19:58:41.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:41.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:41.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:41.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:41.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:41.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=46837' 2026-02-22T19:58:41.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:41.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46837 2026-02-22T19:58:41.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 46837 2026-02-22T19:58:41.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:41.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:41.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:42.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:42.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:42.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:42.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:42.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:42.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:42.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:42.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:42.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:42.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:42.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:42.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:42.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:42.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:42.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:42.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:42.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:42.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:42.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:42.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:42.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:42.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:42.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:42.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:42.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:42.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:42.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:42.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:42.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:42.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:42.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:42.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:42.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:42.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=47069], [object_number=160, tag_tid=11, entry_tid=47068], [object_number=163, tag_tid=11, entry_tid=47067], [object_number=162, tag_tid=11, entry_tid=47066]]], state=connected] 2026-02-22T19:58:42.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:42.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:42.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:42.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:42.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:42.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:42.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:42.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:42.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=47069], [object_number=160, tag_tid=11, entry_tid=47068], [object_number=163, tag_tid=11, entry_tid=47067], [object_number=162, tag_tid=11, entry_tid=47066]]], state=connected] 2026-02-22T19:58:42.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=161, tag_tid=11, entry_tid=47069' 2026-02-22T19:58:42.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=161, tag_tid=11, entry_tid=47069' 2026-02-22T19:58:42.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=47069' '!=' 'object_number=161, tag_tid=11, entry_tid=46837' 2026-02-22T19:58:42.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:42.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=47069' = 'object_number=161, tag_tid=11, entry_tid=46837' 2026-02-22T19:58:42.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=161, tag_tid=11, entry_tid=47069' 2026-02-22T19:58:42.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:42.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:42.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:42.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=47069' 2026-02-22T19:58:42.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:42.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:42.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:42.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:42.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:42.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=47069' 2026-02-22T19:58:42.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:42.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47069 2026-02-22T19:58:42.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 47069 2026-02-22T19:58:42.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:42.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:42.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:42.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:42.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:42.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:42.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:42.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:42.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:42.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:42.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:42.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:42.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:42.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:42.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:42.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:42.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:42.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:42.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:42.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:42.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:42.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=47289], [object_number=160, tag_tid=11, entry_tid=47288], [object_number=163, tag_tid=11, entry_tid=47287], [object_number=162, tag_tid=11, entry_tid=47286]]], state=connected] 2026-02-22T19:58:42.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:42.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:42.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:42.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:42.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:42.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:42.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:42.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=161, tag_tid=11, entry_tid=47289], [object_number=160, tag_tid=11, entry_tid=47288], [object_number=163, tag_tid=11, entry_tid=47287], [object_number=162, tag_tid=11, entry_tid=47286]]], state=connected] 2026-02-22T19:58:42.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=161, tag_tid=11, entry_tid=47289' 2026-02-22T19:58:42.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=161, tag_tid=11, entry_tid=47289' 2026-02-22T19:58:42.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=47289' '!=' 'object_number=161, tag_tid=11, entry_tid=47069' 2026-02-22T19:58:42.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:42.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=161, tag_tid=11, entry_tid=47289' = 'object_number=161, tag_tid=11, entry_tid=47069' 2026-02-22T19:58:42.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=161, tag_tid=11, entry_tid=47289' 2026-02-22T19:58:42.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:42.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=47289' 2026-02-22T19:58:42.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:42.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=161, tag_tid=11, entry_tid=47289' 2026-02-22T19:58:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47289 2026-02-22T19:58:42.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 47289 2026-02-22T19:58:42.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:42.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:42.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:42.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:42.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:42.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:42.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:42.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:42.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:42.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:42.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:42.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:42.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:42.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:42.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:42.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:42.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:42.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:42.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:42.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:42.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:42.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:42.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:42.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:42.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:42.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:42.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:42.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:42.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:42.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:42.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:42.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:42.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:42.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:42.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:42.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:42.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:42.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:42.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:42.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:42.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:42.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:42.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:42.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:42.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=160, tag_tid=11, entry_tid=47480], [object_number=163, tag_tid=11, entry_tid=47479], [object_number=162, tag_tid=11, entry_tid=47478], [object_number=161, tag_tid=11, entry_tid=47477]]], state=connected] 2026-02-22T19:58:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:42.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:42.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:42.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:42.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:42.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=160, tag_tid=11, entry_tid=47480], [object_number=163, tag_tid=11, entry_tid=47479], [object_number=162, tag_tid=11, entry_tid=47478], [object_number=161, tag_tid=11, entry_tid=47477]]], state=connected] 2026-02-22T19:58:42.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=160, tag_tid=11, entry_tid=47480' 2026-02-22T19:58:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=160, tag_tid=11, entry_tid=47480' 2026-02-22T19:58:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=160, tag_tid=11, entry_tid=47480' '!=' 'object_number=161, tag_tid=11, entry_tid=47289' 2026-02-22T19:58:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=160, tag_tid=11, entry_tid=47480' = 'object_number=161, tag_tid=11, entry_tid=47289' 2026-02-22T19:58:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=160, tag_tid=11, entry_tid=47480' 2026-02-22T19:58:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:42.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=160, tag_tid=11, entry_tid=47480' 2026-02-22T19:58:42.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:42.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:42.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:42.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:42.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:42.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=160, tag_tid=11, entry_tid=47480' 2026-02-22T19:58:42.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:42.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:42.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47480 2026-02-22T19:58:42.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 47480 2026-02-22T19:58:42.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:42.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:42.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:43.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:43.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:43.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:43.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:43.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:43.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:43.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:43.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:43.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:43.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:43.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:43.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:43.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=160, tag_tid=11, entry_tid=47728], [object_number=163, tag_tid=11, entry_tid=47727], [object_number=162, tag_tid=11, entry_tid=47726], [object_number=161, tag_tid=11, entry_tid=47725]]], state=connected] 2026-02-22T19:58:43.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:43.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:43.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:43.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:43.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:43.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:43.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=160, tag_tid=11, entry_tid=47728], [object_number=163, tag_tid=11, entry_tid=47727], [object_number=162, tag_tid=11, entry_tid=47726], [object_number=161, tag_tid=11, entry_tid=47725]]], state=connected] 2026-02-22T19:58:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=160, tag_tid=11, entry_tid=47728' 2026-02-22T19:58:43.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=160, tag_tid=11, entry_tid=47728' 2026-02-22T19:58:43.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=160, tag_tid=11, entry_tid=47728' '!=' 'object_number=160, tag_tid=11, entry_tid=47480' 2026-02-22T19:58:43.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:43.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=160, tag_tid=11, entry_tid=47728' = 'object_number=160, tag_tid=11, entry_tid=47480' 2026-02-22T19:58:43.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=160, tag_tid=11, entry_tid=47728' 2026-02-22T19:58:43.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:43.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:43.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:43.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=160, tag_tid=11, entry_tid=47728' 2026-02-22T19:58:43.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:43.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:43.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:43.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:43.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:43.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=160, tag_tid=11, entry_tid=47728' 2026-02-22T19:58:43.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:43.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47728 2026-02-22T19:58:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 47728 2026-02-22T19:58:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:43.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:43.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:43.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:43.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:43.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:43.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:43.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:43.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:43.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:43.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:43.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:43.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:43.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:43.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:43.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:43.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:43.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:43.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:43.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:43.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:43.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:43.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:43.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:43.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:43.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:43.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:43.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:43.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:43.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=47959], [object_number=162, tag_tid=11, entry_tid=47958], [object_number=161, tag_tid=11, entry_tid=47957], [object_number=160, tag_tid=11, entry_tid=47956]]], state=connected] 2026-02-22T19:58:43.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:43.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:43.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:43.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:43.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:43.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:43.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:43.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:43.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:43.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=47959], [object_number=162, tag_tid=11, entry_tid=47958], [object_number=161, tag_tid=11, entry_tid=47957], [object_number=160, tag_tid=11, entry_tid=47956]]], state=connected] 2026-02-22T19:58:43.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=163, tag_tid=11, entry_tid=47959' 2026-02-22T19:58:43.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=163, tag_tid=11, entry_tid=47959' 2026-02-22T19:58:43.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=47959' '!=' 'object_number=160, tag_tid=11, entry_tid=47728' 2026-02-22T19:58:43.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:43.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=47959' = 'object_number=160, tag_tid=11, entry_tid=47728' 2026-02-22T19:58:43.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=163, tag_tid=11, entry_tid=47959' 2026-02-22T19:58:43.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:43.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:43.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:43.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=47959' 2026-02-22T19:58:43.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:43.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:43.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:43.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=47959' 2026-02-22T19:58:43.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:43.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47959 2026-02-22T19:58:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 47959 2026-02-22T19:58:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:43.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:43.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:43.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:43.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:43.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:43.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:43.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:43.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:43.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:43.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:43.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:43.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:43.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:43.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:43.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:43.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:43.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:43.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=48171], [object_number=162, tag_tid=11, entry_tid=48170], [object_number=161, tag_tid=11, entry_tid=48169], [object_number=160, tag_tid=11, entry_tid=48168]]], state=connected] 2026-02-22T19:58:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:43.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:43.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 40 2026-02-22T19:58:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=163, tag_tid=11, entry_tid=48171], [object_number=162, tag_tid=11, entry_tid=48170], [object_number=161, tag_tid=11, entry_tid=48169], [object_number=160, tag_tid=11, entry_tid=48168]]], state=connected] 2026-02-22T19:58:43.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=163, tag_tid=11, entry_tid=48171' 2026-02-22T19:58:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=163, tag_tid=11, entry_tid=48171' 2026-02-22T19:58:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=48171' '!=' 'object_number=163, tag_tid=11, entry_tid=47959' 2026-02-22T19:58:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:43.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=163, tag_tid=11, entry_tid=48171' = 'object_number=163, tag_tid=11, entry_tid=47959' 2026-02-22T19:58:43.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=163, tag_tid=11, entry_tid=48171' 2026-02-22T19:58:43.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:43.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:43.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:43.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=48171' 2026-02-22T19:58:43.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:43.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:43.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:43.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:43.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:43.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=163, tag_tid=11, entry_tid=48171' 2026-02-22T19:58:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:44.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=48171 2026-02-22T19:58:44.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 48171 2026-02-22T19:58:44.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:44.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:44.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:44.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:44.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:44.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:44.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:44.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:44.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:44.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:44.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:44.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:44.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:44.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:44.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:44.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:44.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:44.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:44.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:44.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:44.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:44.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:44.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:44.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:44.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:44.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:44.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:44.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:44.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:44.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=164, tag_tid=11, entry_tid=48316], [object_number=167, tag_tid=11, entry_tid=48315], [object_number=166, tag_tid=11, entry_tid=48314], [object_number=165, tag_tid=11, entry_tid=48313]]], state=connected] 2026-02-22T19:58:44.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:44.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:44.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:44.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:44.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:44.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=164, tag_tid=11, entry_tid=48316], [object_number=167, tag_tid=11, entry_tid=48315], [object_number=166, tag_tid=11, entry_tid=48314], [object_number=165, tag_tid=11, entry_tid=48313]]], state=connected] 2026-02-22T19:58:44.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=164, tag_tid=11, entry_tid=48316' 2026-02-22T19:58:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=164, tag_tid=11, entry_tid=48316' 2026-02-22T19:58:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=164, tag_tid=11, entry_tid=48316' '!=' 'object_number=163, tag_tid=11, entry_tid=48171' 2026-02-22T19:58:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=164, tag_tid=11, entry_tid=48316' = 'object_number=163, tag_tid=11, entry_tid=48171' 2026-02-22T19:58:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=164, tag_tid=11, entry_tid=48316' 2026-02-22T19:58:44.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:44.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:44.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:44.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:44.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=164, tag_tid=11, entry_tid=48316' 2026-02-22T19:58:44.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:44.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:44.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:44.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:44.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:44.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:44.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=164, tag_tid=11, entry_tid=48316' 2026-02-22T19:58:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:44.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=48316 2026-02-22T19:58:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 48316 2026-02-22T19:58:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:44.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:44.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:44.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:44.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:44.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:44.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:44.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:44.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:44.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:44.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:44.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:44.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=167, tag_tid=11, entry_tid=48531], [object_number=166, tag_tid=11, entry_tid=48530], [object_number=165, tag_tid=11, entry_tid=48529], [object_number=164, tag_tid=11, entry_tid=48528]]], state=connected] 2026-02-22T19:58:44.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:44.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:44.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:44.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:44.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=167, tag_tid=11, entry_tid=48531], [object_number=166, tag_tid=11, entry_tid=48530], [object_number=165, tag_tid=11, entry_tid=48529], [object_number=164, tag_tid=11, entry_tid=48528]]], state=connected] 2026-02-22T19:58:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=167, tag_tid=11, entry_tid=48531' 2026-02-22T19:58:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=167, tag_tid=11, entry_tid=48531' 2026-02-22T19:58:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=167, tag_tid=11, entry_tid=48531' '!=' 'object_number=164, tag_tid=11, entry_tid=48316' 2026-02-22T19:58:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=167, tag_tid=11, entry_tid=48531' = 'object_number=164, tag_tid=11, entry_tid=48316' 2026-02-22T19:58:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=167, tag_tid=11, entry_tid=48531' 2026-02-22T19:58:44.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:44.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:44.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:44.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:44.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=167, tag_tid=11, entry_tid=48531' 2026-02-22T19:58:44.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:44.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:44.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:44.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:44.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:44.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:44.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:44.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=167, tag_tid=11, entry_tid=48531' 2026-02-22T19:58:44.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:44.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:44.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=48531 2026-02-22T19:58:44.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 48531 2026-02-22T19:58:44.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:44.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:44.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:44.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:44.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:44.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:44.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:44.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:44.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:45.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:45.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:45.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:45.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:45.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:45.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:45.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=48718], [object_number=165, tag_tid=11, entry_tid=48717], [object_number=164, tag_tid=11, entry_tid=48716], [object_number=167, tag_tid=11, entry_tid=48715]]], state=connected] 2026-02-22T19:58:45.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:45.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:45.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=48718], [object_number=165, tag_tid=11, entry_tid=48717], [object_number=164, tag_tid=11, entry_tid=48716], [object_number=167, tag_tid=11, entry_tid=48715]]], state=connected] 2026-02-22T19:58:45.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=166, tag_tid=11, entry_tid=48718' 2026-02-22T19:58:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=166, tag_tid=11, entry_tid=48718' 2026-02-22T19:58:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=48718' '!=' 'object_number=167, tag_tid=11, entry_tid=48531' 2026-02-22T19:58:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=48718' = 'object_number=167, tag_tid=11, entry_tid=48531' 2026-02-22T19:58:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=166, tag_tid=11, entry_tid=48718' 2026-02-22T19:58:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:45.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=48718' 2026-02-22T19:58:45.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:45.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:45.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:45.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:45.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=48718' 2026-02-22T19:58:45.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:45.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=48718 2026-02-22T19:58:45.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 48718 2026-02-22T19:58:45.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:45.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:45.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:45.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:45.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:45.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:45.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:45.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:45.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:45.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:45.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:45.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:45.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=167, tag_tid=11, entry_tid=48935], [object_number=166, tag_tid=11, entry_tid=48934], [object_number=165, tag_tid=11, entry_tid=48933], [object_number=164, tag_tid=11, entry_tid=48932]]], state=connected] 2026-02-22T19:58:45.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:45.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:45.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=167, tag_tid=11, entry_tid=48935], [object_number=166, tag_tid=11, entry_tid=48934], [object_number=165, tag_tid=11, entry_tid=48933], [object_number=164, tag_tid=11, entry_tid=48932]]], state=connected] 2026-02-22T19:58:45.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=167, tag_tid=11, entry_tid=48935' 2026-02-22T19:58:45.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=167, tag_tid=11, entry_tid=48935' 2026-02-22T19:58:45.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=167, tag_tid=11, entry_tid=48935' '!=' 'object_number=166, tag_tid=11, entry_tid=48718' 2026-02-22T19:58:45.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:45.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=167, tag_tid=11, entry_tid=48935' = 'object_number=166, tag_tid=11, entry_tid=48718' 2026-02-22T19:58:45.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=167, tag_tid=11, entry_tid=48935' 2026-02-22T19:58:45.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:45.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:45.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:45.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=167, tag_tid=11, entry_tid=48935' 2026-02-22T19:58:45.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:45.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=167, tag_tid=11, entry_tid=48935' 2026-02-22T19:58:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=48935 2026-02-22T19:58:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 48935 2026-02-22T19:58:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:45.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:45.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:45.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:45.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:45.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:45.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:45.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:45.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:45.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:45.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:45.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:45.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:45.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=49142], [object_number=165, tag_tid=11, entry_tid=49141], [object_number=164, tag_tid=11, entry_tid=49140], [object_number=167, tag_tid=11, entry_tid=49139]]], state=connected] 2026-02-22T19:58:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:45.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:45.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:45.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:45.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:45.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:45.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=49142], [object_number=165, tag_tid=11, entry_tid=49141], [object_number=164, tag_tid=11, entry_tid=49140], [object_number=167, tag_tid=11, entry_tid=49139]]], state=connected] 2026-02-22T19:58:45.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=166, tag_tid=11, entry_tid=49142' 2026-02-22T19:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=166, tag_tid=11, entry_tid=49142' 2026-02-22T19:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=49142' '!=' 'object_number=167, tag_tid=11, entry_tid=48935' 2026-02-22T19:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:45.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=49142' = 'object_number=167, tag_tid=11, entry_tid=48935' 2026-02-22T19:58:45.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=166, tag_tid=11, entry_tid=49142' 2026-02-22T19:58:45.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:45.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:45.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:45.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=49142' 2026-02-22T19:58:45.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:45.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:45.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:45.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:45.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:45.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=49142' 2026-02-22T19:58:45.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:45.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:45.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=49142 2026-02-22T19:58:45.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 49142 2026-02-22T19:58:45.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:45.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:45.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:46.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:46.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:46.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:46.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:46.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:46.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:46.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:46.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:46.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:46.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:46.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:46.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:46.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:46.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=49366], [object_number=165, tag_tid=11, entry_tid=49365], [object_number=164, tag_tid=11, entry_tid=49364], [object_number=167, tag_tid=11, entry_tid=49363]]], state=connected] 2026-02-22T19:58:46.178 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6021037933661365) > weight (0.25) 2026-02-22T19:58:46.178 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.38138343145581977) > weight (0.25) 2026-02-22T19:58:46.178 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3936787825729233) > weight (0.25) 2026-02-22T19:58:46.178 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6403310554884151) > weight (0.25) 2026-02-22T19:58:46.179 INFO:tasks.rbd_mirror_thrash:waiting for 63.70545751001114 secs before thrashing 2026-02-22T19:58:46.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:46.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:46.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:46.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:46.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=49366], [object_number=165, tag_tid=11, entry_tid=49365], [object_number=164, tag_tid=11, entry_tid=49364], [object_number=167, tag_tid=11, entry_tid=49363]]], state=connected] 2026-02-22T19:58:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=166, tag_tid=11, entry_tid=49366' 2026-02-22T19:58:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=166, tag_tid=11, entry_tid=49366' 2026-02-22T19:58:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=49366' '!=' 'object_number=166, tag_tid=11, entry_tid=49142' 2026-02-22T19:58:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=49366' = 'object_number=166, tag_tid=11, entry_tid=49142' 2026-02-22T19:58:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=166, tag_tid=11, entry_tid=49366' 2026-02-22T19:58:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:46.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:46.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:46.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=49366' 2026-02-22T19:58:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:46.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:46.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:46.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=49366' 2026-02-22T19:58:46.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:46.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=49366 2026-02-22T19:58:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 49366 2026-02-22T19:58:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:46.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:46.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:46.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:46.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:46.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:46.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:46.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:46.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:46.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=49577], [object_number=164, tag_tid=11, entry_tid=49576], [object_number=167, tag_tid=11, entry_tid=49575], [object_number=166, tag_tid=11, entry_tid=49574]]], state=connected] 2026-02-22T19:58:46.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:46.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:46.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:46.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:46.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:46.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:46.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=49577], [object_number=164, tag_tid=11, entry_tid=49576], [object_number=167, tag_tid=11, entry_tid=49575], [object_number=166, tag_tid=11, entry_tid=49574]]], state=connected] 2026-02-22T19:58:46.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=11, entry_tid=49577' 2026-02-22T19:58:46.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=11, entry_tid=49577' 2026-02-22T19:58:46.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=49577' '!=' 'object_number=166, tag_tid=11, entry_tid=49366' 2026-02-22T19:58:46.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:46.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=49577' = 'object_number=166, tag_tid=11, entry_tid=49366' 2026-02-22T19:58:46.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=165, tag_tid=11, entry_tid=49577' 2026-02-22T19:58:46.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:46.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:46.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:46.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:46.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=49577' 2026-02-22T19:58:46.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:46.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:46.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:46.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:46.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:46.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:46.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=49577' 2026-02-22T19:58:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=49577 2026-02-22T19:58:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 49577 2026-02-22T19:58:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:46.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:46.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:46.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:46.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:46.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:46.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:46.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:46.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:46.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:46.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:46.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:46.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:47.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:47.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:47.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:47.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=49793], [object_number=164, tag_tid=11, entry_tid=49792], [object_number=167, tag_tid=11, entry_tid=49791], [object_number=166, tag_tid=11, entry_tid=49790]]], state=connected] 2026-02-22T19:58:47.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=49793], [object_number=164, tag_tid=11, entry_tid=49792], [object_number=167, tag_tid=11, entry_tid=49791], [object_number=166, tag_tid=11, entry_tid=49790]]], state=connected] 2026-02-22T19:58:47.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=11, entry_tid=49793' 2026-02-22T19:58:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=11, entry_tid=49793' 2026-02-22T19:58:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=49793' '!=' 'object_number=165, tag_tid=11, entry_tid=49577' 2026-02-22T19:58:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=49793' = 'object_number=165, tag_tid=11, entry_tid=49577' 2026-02-22T19:58:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=165, tag_tid=11, entry_tid=49793' 2026-02-22T19:58:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:47.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=49793' 2026-02-22T19:58:47.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:47.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:47.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=49793' 2026-02-22T19:58:47.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:47.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=49793 2026-02-22T19:58:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 49793 2026-02-22T19:58:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:47.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:47.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:47.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:47.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:47.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:47.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:47.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:47.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:47.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:47.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:47.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:47.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:47.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=50117], [object_number=164, tag_tid=11, entry_tid=50116], [object_number=167, tag_tid=11, entry_tid=50115], [object_number=166, tag_tid=11, entry_tid=50114]]], state=connected] 2026-02-22T19:58:47.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:47.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:47.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:47.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:47.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:47.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:47.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:47.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:47.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=50117], [object_number=164, tag_tid=11, entry_tid=50116], [object_number=167, tag_tid=11, entry_tid=50115], [object_number=166, tag_tid=11, entry_tid=50114]]], state=connected] 2026-02-22T19:58:47.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=11, entry_tid=50117' 2026-02-22T19:58:47.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=11, entry_tid=50117' 2026-02-22T19:58:47.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=50117' '!=' 'object_number=165, tag_tid=11, entry_tid=49793' 2026-02-22T19:58:47.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:47.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=50117' = 'object_number=165, tag_tid=11, entry_tid=49793' 2026-02-22T19:58:47.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=165, tag_tid=11, entry_tid=50117' 2026-02-22T19:58:47.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:47.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:47.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:47.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=50117' 2026-02-22T19:58:47.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:47.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:47.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=50117' 2026-02-22T19:58:47.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=50117 2026-02-22T19:58:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 50117 2026-02-22T19:58:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:47.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:47.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:47.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:47.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:47.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:47.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:47.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:47.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:47.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:47.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:47.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:47.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:47.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:47.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:47.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:47.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:47.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:47.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:47.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:47.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:47.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:47.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:47.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=50322], [object_number=165, tag_tid=11, entry_tid=50321], [object_number=164, tag_tid=11, entry_tid=50320], [object_number=167, tag_tid=11, entry_tid=50319]]], state=connected] 2026-02-22T19:58:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:47.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:47.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:47.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:47.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:47.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=50322], [object_number=165, tag_tid=11, entry_tid=50321], [object_number=164, tag_tid=11, entry_tid=50320], [object_number=167, tag_tid=11, entry_tid=50319]]], state=connected] 2026-02-22T19:58:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=166, tag_tid=11, entry_tid=50322' 2026-02-22T19:58:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=166, tag_tid=11, entry_tid=50322' 2026-02-22T19:58:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=50322' '!=' 'object_number=165, tag_tid=11, entry_tid=50117' 2026-02-22T19:58:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=50322' = 'object_number=165, tag_tid=11, entry_tid=50117' 2026-02-22T19:58:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=166, tag_tid=11, entry_tid=50322' 2026-02-22T19:58:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=50322' 2026-02-22T19:58:47.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:47.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=50322' 2026-02-22T19:58:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=50322 2026-02-22T19:58:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 50322 2026-02-22T19:58:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:48.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:48.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:48.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:48.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:48.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:48.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:48.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:48.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:48.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:48.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:48.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:48.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=50534], [object_number=165, tag_tid=11, entry_tid=50533], [object_number=164, tag_tid=11, entry_tid=50532], [object_number=167, tag_tid=11, entry_tid=50531]]], state=connected] 2026-02-22T19:58:48.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=166, tag_tid=11, entry_tid=50534], [object_number=165, tag_tid=11, entry_tid=50533], [object_number=164, tag_tid=11, entry_tid=50532], [object_number=167, tag_tid=11, entry_tid=50531]]], state=connected] 2026-02-22T19:58:48.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=166, tag_tid=11, entry_tid=50534' 2026-02-22T19:58:48.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=166, tag_tid=11, entry_tid=50534' 2026-02-22T19:58:48.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=50534' '!=' 'object_number=166, tag_tid=11, entry_tid=50322' 2026-02-22T19:58:48.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:48.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=166, tag_tid=11, entry_tid=50534' = 'object_number=166, tag_tid=11, entry_tid=50322' 2026-02-22T19:58:48.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=166, tag_tid=11, entry_tid=50534' 2026-02-22T19:58:48.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:48.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:48.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:48.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:48.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=50534' 2026-02-22T19:58:48.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:48.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:48.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:48.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:48.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:48.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:48.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:48.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=166, tag_tid=11, entry_tid=50534' 2026-02-22T19:58:48.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:48.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:48.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=50534 2026-02-22T19:58:48.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 50534 2026-02-22T19:58:48.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:48.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:48.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:48.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:48.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:48.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:48.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:48.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:48.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:48.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:48.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:48.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:48.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:48.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:48.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:48.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:48.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:48.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:48.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:48.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:48.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:48.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:48.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:48.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:48.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:48.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:48.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:48.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:48.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:48.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:48.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:48.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:48.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:48.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:48.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:48.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:48.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:48.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:48.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:48.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:48.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:48.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:48.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:48.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=167, tag_tid=11, entry_tid=50731], [object_number=166, tag_tid=11, entry_tid=50730], [object_number=165, tag_tid=11, entry_tid=50729], [object_number=164, tag_tid=11, entry_tid=50728]]], state=connected] 2026-02-22T19:58:48.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:48.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:48.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:48.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:48.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:49.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:49.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:49.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:49.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=167, tag_tid=11, entry_tid=50731], [object_number=166, tag_tid=11, entry_tid=50730], [object_number=165, tag_tid=11, entry_tid=50729], [object_number=164, tag_tid=11, entry_tid=50728]]], state=connected] 2026-02-22T19:58:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=167, tag_tid=11, entry_tid=50731' 2026-02-22T19:58:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=167, tag_tid=11, entry_tid=50731' 2026-02-22T19:58:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=167, tag_tid=11, entry_tid=50731' '!=' 'object_number=166, tag_tid=11, entry_tid=50534' 2026-02-22T19:58:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=167, tag_tid=11, entry_tid=50731' = 'object_number=166, tag_tid=11, entry_tid=50534' 2026-02-22T19:58:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=167, tag_tid=11, entry_tid=50731' 2026-02-22T19:58:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=167, tag_tid=11, entry_tid=50731' 2026-02-22T19:58:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:49.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=167, tag_tid=11, entry_tid=50731' 2026-02-22T19:58:49.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:49.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=50731 2026-02-22T19:58:49.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 50731 2026-02-22T19:58:49.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:49.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:49.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:49.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:49.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:49.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:49.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:49.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:49.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:49.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:49.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:49.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:49.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:49.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:49.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:49.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:49.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:49.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:49.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:49.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:49.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:49.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:49.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:49.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:49.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:49.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:49.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:49.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:49.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51321], [object_number=164, tag_tid=11, entry_tid=51320], [object_number=167, tag_tid=11, entry_tid=51319], [object_number=166, tag_tid=11, entry_tid=51318]]], state=connected] 2026-02-22T19:58:49.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:49.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:49.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:49.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:49.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:49.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:49.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:49.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:49.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:49.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:49.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:49.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:49.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:49.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:49.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:49.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:49.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:49.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:49.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51321], [object_number=164, tag_tid=11, entry_tid=51320], [object_number=167, tag_tid=11, entry_tid=51319], [object_number=166, tag_tid=11, entry_tid=51318]]], state=connected] 2026-02-22T19:58:49.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=11, entry_tid=51321' 2026-02-22T19:58:49.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=11, entry_tid=51321' 2026-02-22T19:58:49.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=51321' '!=' 'object_number=167, tag_tid=11, entry_tid=50731' 2026-02-22T19:58:49.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:49.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=51321' = 'object_number=167, tag_tid=11, entry_tid=50731' 2026-02-22T19:58:49.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=165, tag_tid=11, entry_tid=51321' 2026-02-22T19:58:49.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:49.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:49.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=51321' 2026-02-22T19:58:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:49.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:49.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:49.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:49.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=51321' 2026-02-22T19:58:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=51321 2026-02-22T19:58:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 51321 2026-02-22T19:58:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:49.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:49.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:49.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:49.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:49.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:49.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:49.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:49.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:49.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:49.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:49.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:49.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:49.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=51361' '!=' 'object_number=165, tag_tid=11, entry_tid=51321' 2026-02-22T19:58:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:58:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=51361' = 'object_number=165, tag_tid=11, entry_tid=51321' 2026-02-22T19:58:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=12 2026-02-22T19:58:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:49.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:58:49.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=11 2026-02-22T19:58:49.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:49.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:49.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:58:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:58:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:58:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=51361 2026-02-22T19:58:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 12 = 11 -a 1 -le 51361 2026-02-22T19:58:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:58:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:58:50.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:50.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:50.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:50.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:50.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:50.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:50.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:50.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:50.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:50.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:50.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:50.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:50.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:50.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:50.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:50.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:50.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:50.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:50.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:50.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:50.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:50.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:50.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:50.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:50.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:50.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:50.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:50.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:50.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:50.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:50.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:50.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:50.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:50.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:50.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:50.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:50.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:50.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:50.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:50.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:50.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:50.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:50.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:50.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:50.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:50.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:50.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:50.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:50.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:50.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:50.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:50.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:50.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:50.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=51361' '!=' 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:50.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:50.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T19:58:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:50.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:50.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:50.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:50.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:50.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:50.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:50.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:50.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:50.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:50.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:50.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:50.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:50.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:50.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:50.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:50.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:50.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:50.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:50.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:50.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:50.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:50.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:50.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:50.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:50.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:50.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:50.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:50.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:50.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:50.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:50.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:50.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:50.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=51361' '!=' 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:51.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:51.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:51.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:51.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:51.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:51.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:51.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:51.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:51.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:51.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:51.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:51.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:51.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:51.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:51.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:51.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=51361' '!=' 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:51.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:51.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T19:58:53.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:53.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:53.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:53.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:53.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:53.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:53.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:53.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:53.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:53.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:53.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:53.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:53.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:53.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:53.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:53.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:53.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:53.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:53.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:53.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:53.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:53.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=11, entry_tid=51361], [object_number=164, tag_tid=11, entry_tid=51360], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:53.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:53.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:53.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=165, tag_tid=11, entry_tid=51361' '!=' 'object_number=165, tag_tid=11, entry_tid=51361' 2026-02-22T19:58:53.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:53.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:58:55.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:55.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:58:55.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:55.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:58:55.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:58:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:55.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:55.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:55.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:55.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:55.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:55.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:55.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:55.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:55.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:58:55.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:55.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:55.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:55.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:58:55.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:58:55.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:58:55.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:58:55.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:58:55.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:55.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:58:55.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:55.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:55.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:58:55.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:55.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 41 2026-02-22T19:58:55.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 41 2026-02-22T19:58:55.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:58:55.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:55.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=165, tag_tid=12, entry_tid=1], [object_number=164, tag_tid=12, entry_tid=0], [object_number=167, tag_tid=11, entry_tid=51359], [object_number=166, tag_tid=11, entry_tid=51358]]], state=connected] 2026-02-22T19:58:55.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:58:55.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:55.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=165, tag_tid=12, entry_tid=1' -a 'object_number=165, tag_tid=12, entry_tid=1' = 'object_number=165, tag_tid=12, entry_tid=1' 2026-02-22T19:58:55.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:55.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2026-02-22T19:58:55.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:58:55.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:55.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:55.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap4 2026-02-22T19:58:55.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:58:55.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T19:58:55.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:58:56.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 7 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 19:57:16 2026 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 19:58:40 2026 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T19:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T19:58:56.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T19:58:56.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T19:58:56.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:58:56.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T19:58:56.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n snap3-clean ']' 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap3-clean 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap3-clean 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:56.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:58:57.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:58:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:58.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:58:58.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:58.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:58.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap4 2026-02-22T19:58:58.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:58.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:58.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap4 2026-02-22T19:58:58.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T19:58:58.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:58:58.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:58.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:58.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:58:58.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:58:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:59.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:58:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:58:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:58:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap4-clean 2026-02-22T19:58:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap4-clean 2026-02-22T19:58:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:58:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:58:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:58:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap4-clean 2026-02-22T19:58:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T19:59:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T19:59:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T19:59:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:59:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T19:59:00.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=8 2026-02-22T19:59:00.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T19:59:00.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T19:59:00.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 628736 bytes 2026-02-22T19:59:01.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 50194432 bytes 2026-02-22T19:59:01.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 71651328 bytes 2026-02-22T19:59:02.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 138915840 bytes 2026-02-22T19:59:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 204569600 bytes 2026-02-22T19:59:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 255198208 bytes 2026-02-22T19:59:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 276529152 bytes 2026-02-22T19:59:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 278071296 bytes 2026-02-22T19:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 317544448 bytes 2026-02-22T19:59:07.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 336438272 bytes 2026-02-22T19:59:07.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 352720896 bytes 2026-02-22T19:59:07.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 386623488 bytes 2026-02-22T19:59:08.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 406097920 bytes 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:59:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:59:08.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:08.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:59:08.905 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5801615761581566) > weight (0.25) 2026-02-22T19:59:08.905 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7830868623778281) > weight (0.25) 2026-02-22T19:59:08.906 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4431200170475246) > weight (0.25) 2026-02-22T19:59:08.906 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6624260459469066) > weight (0.25) 2026-02-22T19:59:08.906 INFO:tasks.rbd_mirror_thrash:waiting for 79.40149767294172 secs before thrashing 2026-02-22T19:59:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:59:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T19:59:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T19:59:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:59:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:59:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:59:09.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T19:59:09.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":5887903.97,"entries_behind_primary":0,"entries_per_second":372.9,"non_primary_position":{"entry_tid":1,"object_number":165,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":165,"tag_tid":12}} 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.0 on trial066 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:59:03 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 19:59:03' 2026-02-22T19:59:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:59:09.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T19:59:09.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T19:59:09.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":5887903.97,"entries_behind_primary":0,"entries_per_second":372.9,"non_primary_position":{"entry_tid":1,"object_number":165,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":165,"tag_tid":12}} 2026-02-22T19:59:09.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T19:59:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:59:09.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T19:59:09.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:59:09.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:09.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.0 on trial066 2026-02-22T19:59:09.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T19:59:09.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:59:03' 2026-02-22T19:59:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:59:03' 2026-02-22T19:59:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 19:59:03' 2026-02-22T19:59:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T19:59:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T19:59:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:09.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142024257mirror.0trial06619.2.3-689-gf90f4eaetrueOK42134266mirror.3trial06619.2.3-689-gf90f4eaefalseOK43154318mirror.1trial06619.2.3-689-gf90f4eaefalseOK43504353mirror.2trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":5887903.97,"entries_behind_primary":0,"entries_per_second":372.9,"non_primary_position":{"entry_tid":1,"object_number":165,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":165,"tag_tid":12}}42024257mirror.0trial0662026-02-22 19:59:03cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 19:59:03' 2026-02-22T19:59:09.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T19:59:09.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:09.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 19:59:03' 2026-02-22T19:59:09.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:09.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T19:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T19:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T19:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T19:59:09.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap5 2026-02-22T19:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap5 2026-02-22T19:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap5 2026-02-22T19:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2026-02-22T19:59:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:59:40.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:40.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:59:41.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:59:41.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:41.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:41.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:41.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T19:59:41.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T19:59:41.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T19:59:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T19:59:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:59:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T19:59:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T19:59:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T19:59:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T19:59:41.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:41.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:41.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:41.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:41.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T19:59:41.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:59:41.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:59:41.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T19:59:41.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T19:59:41.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:59:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T19:59:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T19:59:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T19:59:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T19:59:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:41.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:41.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:41.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:41.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:41.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:41.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:41.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:41.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:41.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:42.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 46 2026-02-22T19:59:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=187, tag_tid=14, entry_tid=20911], [object_number=186, tag_tid=14, entry_tid=20910], [object_number=185, tag_tid=14, entry_tid=20909], [object_number=184, tag_tid=14, entry_tid=20908]]], state=connected] 2026-02-22T19:59:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:42.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:42.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 46 2026-02-22T19:59:42.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:42.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:42.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:42.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=187, tag_tid=14, entry_tid=20911], [object_number=186, tag_tid=14, entry_tid=20910], [object_number=185, tag_tid=14, entry_tid=20909], [object_number=184, tag_tid=14, entry_tid=20908]]], state=connected] 2026-02-22T19:59:42.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=187, tag_tid=14, entry_tid=20911' 2026-02-22T19:59:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=187, tag_tid=14, entry_tid=20911' 2026-02-22T19:59:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=187, tag_tid=14, entry_tid=20911' '!=' '' 2026-02-22T19:59:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=187, tag_tid=14, entry_tid=20911' = '' 2026-02-22T19:59:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=187, tag_tid=14, entry_tid=20911' 2026-02-22T19:59:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=187, tag_tid=14, entry_tid=20911' 2026-02-22T19:59:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:42.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:42.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=187, tag_tid=14, entry_tid=20911' 2026-02-22T19:59:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20911 2026-02-22T19:59:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 20911 2026-02-22T19:59:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:42.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:42.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:42.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:42.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:42.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:42.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:42.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:42.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:42.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:42.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:42.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:42.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 46 2026-02-22T19:59:42.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:42.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:42.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:42.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=186, tag_tid=14, entry_tid=21134], [object_number=185, tag_tid=14, entry_tid=21133], [object_number=184, tag_tid=14, entry_tid=21132], [object_number=187, tag_tid=14, entry_tid=21131]]], state=connected] 2026-02-22T19:59:42.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:42.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:42.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:42.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 46 2026-02-22T19:59:42.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:42.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:42.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:42.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=186, tag_tid=14, entry_tid=21134], [object_number=185, tag_tid=14, entry_tid=21133], [object_number=184, tag_tid=14, entry_tid=21132], [object_number=187, tag_tid=14, entry_tid=21131]]], state=connected] 2026-02-22T19:59:42.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=186, tag_tid=14, entry_tid=21134' 2026-02-22T19:59:42.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=186, tag_tid=14, entry_tid=21134' 2026-02-22T19:59:42.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=186, tag_tid=14, entry_tid=21134' '!=' 'object_number=187, tag_tid=14, entry_tid=20911' 2026-02-22T19:59:42.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:42.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=186, tag_tid=14, entry_tid=21134' = 'object_number=187, tag_tid=14, entry_tid=20911' 2026-02-22T19:59:42.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=186, tag_tid=14, entry_tid=21134' 2026-02-22T19:59:42.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:42.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:42.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:42.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=186, tag_tid=14, entry_tid=21134' 2026-02-22T19:59:42.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:42.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:42.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:42.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:42.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:42.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=186, tag_tid=14, entry_tid=21134' 2026-02-22T19:59:42.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:42.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21134 2026-02-22T19:59:42.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 21134 2026-02-22T19:59:42.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:42.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:42.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:42.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:42.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:42.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:42.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:42.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:42.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:42.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:42.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:42.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:42.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:42.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:42.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:42.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:42.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:42.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:42.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:42.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:42.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:42.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:42.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:42.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 46 2026-02-22T19:59:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=185, tag_tid=14, entry_tid=21345], [object_number=184, tag_tid=14, entry_tid=21344], [object_number=187, tag_tid=14, entry_tid=21343], [object_number=186, tag_tid=14, entry_tid=21342]]], state=connected] 2026-02-22T19:59:42.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:42.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:42.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:42.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:42.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:42.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:42.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:42.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:42.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:42.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:42.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:42.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 46 2026-02-22T19:59:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=185, tag_tid=14, entry_tid=21345], [object_number=184, tag_tid=14, entry_tid=21344], [object_number=187, tag_tid=14, entry_tid=21343], [object_number=186, tag_tid=14, entry_tid=21342]]], state=connected] 2026-02-22T19:59:42.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:42.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=185, tag_tid=14, entry_tid=21345' 2026-02-22T19:59:42.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=185, tag_tid=14, entry_tid=21345' 2026-02-22T19:59:42.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=185, tag_tid=14, entry_tid=21345' '!=' 'object_number=186, tag_tid=14, entry_tid=21134' 2026-02-22T19:59:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=185, tag_tid=14, entry_tid=21345' = 'object_number=186, tag_tid=14, entry_tid=21134' 2026-02-22T19:59:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=185, tag_tid=14, entry_tid=21345' 2026-02-22T19:59:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=185, tag_tid=14, entry_tid=21345' 2026-02-22T19:59:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:42.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:42.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:42.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=185, tag_tid=14, entry_tid=21345' 2026-02-22T19:59:42.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:42.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:42.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21345 2026-02-22T19:59:42.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 21345 2026-02-22T19:59:42.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:42.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:42.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:43.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:43.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:43.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:43.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:43.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:43.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:43.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:43.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:43.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 46 2026-02-22T19:59:43.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:43.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:43.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:43.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=187, tag_tid=14, entry_tid=21559], [object_number=186, tag_tid=14, entry_tid=21558], [object_number=185, tag_tid=14, entry_tid=21557], [object_number=184, tag_tid=14, entry_tid=21556]]], state=connected] 2026-02-22T19:59:43.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:43.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:43.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 46 2026-02-22T19:59:43.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:43.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:43.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:43.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=187, tag_tid=14, entry_tid=21559], [object_number=186, tag_tid=14, entry_tid=21558], [object_number=185, tag_tid=14, entry_tid=21557], [object_number=184, tag_tid=14, entry_tid=21556]]], state=connected] 2026-02-22T19:59:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=187, tag_tid=14, entry_tid=21559' 2026-02-22T19:59:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=187, tag_tid=14, entry_tid=21559' 2026-02-22T19:59:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=187, tag_tid=14, entry_tid=21559' '!=' 'object_number=185, tag_tid=14, entry_tid=21345' 2026-02-22T19:59:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=187, tag_tid=14, entry_tid=21559' = 'object_number=185, tag_tid=14, entry_tid=21345' 2026-02-22T19:59:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=187, tag_tid=14, entry_tid=21559' 2026-02-22T19:59:43.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:43.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:43.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:43.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:43.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=187, tag_tid=14, entry_tid=21559' 2026-02-22T19:59:43.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:43.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:43.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:43.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:43.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:43.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:43.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=187, tag_tid=14, entry_tid=21559' 2026-02-22T19:59:43.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:43.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:43.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21559 2026-02-22T19:59:43.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 21559 2026-02-22T19:59:43.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:43.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:43.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:43.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:43.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:43.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:43.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:43.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:43.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:43.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:43.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:43.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:43.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:43.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:43.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:43.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:43.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:43.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:43.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:43.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:43.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:43.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:43.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:43.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=21798], [object_number=189, tag_tid=14, entry_tid=21797], [object_number=188, tag_tid=14, entry_tid=21796], [object_number=191, tag_tid=14, entry_tid=21795]]], state=connected] 2026-02-22T19:59:43.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:43.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:43.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:43.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:43.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:43.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:43.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:43.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:43.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=21798], [object_number=189, tag_tid=14, entry_tid=21797], [object_number=188, tag_tid=14, entry_tid=21796], [object_number=191, tag_tid=14, entry_tid=21795]]], state=connected] 2026-02-22T19:59:43.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=190, tag_tid=14, entry_tid=21798' 2026-02-22T19:59:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=190, tag_tid=14, entry_tid=21798' 2026-02-22T19:59:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=21798' '!=' 'object_number=187, tag_tid=14, entry_tid=21559' 2026-02-22T19:59:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=21798' = 'object_number=187, tag_tid=14, entry_tid=21559' 2026-02-22T19:59:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=190, tag_tid=14, entry_tid=21798' 2026-02-22T19:59:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:43.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:43.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=21798' 2026-02-22T19:59:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:43.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:43.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:43.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:43.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:43.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:43.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=21798' 2026-02-22T19:59:43.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:43.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:43.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21798 2026-02-22T19:59:43.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 21798 2026-02-22T19:59:43.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:43.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:43.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:44.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:44.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:44.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:44.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:44.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:44.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:44.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:44.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:44.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:44.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:44.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:44.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=22008], [object_number=191, tag_tid=14, entry_tid=22007], [object_number=190, tag_tid=14, entry_tid=22006], [object_number=189, tag_tid=14, entry_tid=22005]]], state=connected] 2026-02-22T19:59:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:44.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:44.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=22008], [object_number=191, tag_tid=14, entry_tid=22007], [object_number=190, tag_tid=14, entry_tid=22006], [object_number=189, tag_tid=14, entry_tid=22005]]], state=connected] 2026-02-22T19:59:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=188, tag_tid=14, entry_tid=22008' 2026-02-22T19:59:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=188, tag_tid=14, entry_tid=22008' 2026-02-22T19:59:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=22008' '!=' 'object_number=190, tag_tid=14, entry_tid=21798' 2026-02-22T19:59:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=22008' = 'object_number=190, tag_tid=14, entry_tid=21798' 2026-02-22T19:59:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=188, tag_tid=14, entry_tid=22008' 2026-02-22T19:59:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:44.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=22008' 2026-02-22T19:59:44.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:44.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:44.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:44.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:44.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:44.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=22008' 2026-02-22T19:59:44.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:44.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22008 2026-02-22T19:59:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 22008 2026-02-22T19:59:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:44.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:44.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:44.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:44.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:44.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:44.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:44.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:44.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:44.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:44.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:44.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:44.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:44.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:44.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:44.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:44.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:44.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:44.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=191, tag_tid=14, entry_tid=22219], [object_number=190, tag_tid=14, entry_tid=22218], [object_number=189, tag_tid=14, entry_tid=22217], [object_number=188, tag_tid=14, entry_tid=22216]]], state=connected] 2026-02-22T19:59:44.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:44.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=191, tag_tid=14, entry_tid=22219], [object_number=190, tag_tid=14, entry_tid=22218], [object_number=189, tag_tid=14, entry_tid=22217], [object_number=188, tag_tid=14, entry_tid=22216]]], state=connected] 2026-02-22T19:59:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=191, tag_tid=14, entry_tid=22219' 2026-02-22T19:59:44.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=191, tag_tid=14, entry_tid=22219' 2026-02-22T19:59:44.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=191, tag_tid=14, entry_tid=22219' '!=' 'object_number=188, tag_tid=14, entry_tid=22008' 2026-02-22T19:59:44.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:44.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=191, tag_tid=14, entry_tid=22219' = 'object_number=188, tag_tid=14, entry_tid=22008' 2026-02-22T19:59:44.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=191, tag_tid=14, entry_tid=22219' 2026-02-22T19:59:44.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:44.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:44.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:44.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=191, tag_tid=14, entry_tid=22219' 2026-02-22T19:59:44.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:44.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:44.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:44.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=191, tag_tid=14, entry_tid=22219' 2026-02-22T19:59:44.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:44.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22219 2026-02-22T19:59:44.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 22219 2026-02-22T19:59:44.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:44.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:44.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:44.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:44.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:44.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:44.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:44.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:44.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:44.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:44.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:44.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:44.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:44.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:44.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:44.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:44.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:44.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=22424], [object_number=191, tag_tid=14, entry_tid=22423], [object_number=190, tag_tid=14, entry_tid=22422], [object_number=189, tag_tid=14, entry_tid=22421]]], state=connected] 2026-02-22T19:59:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:44.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:44.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:44.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:44.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:44.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:44.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=22424], [object_number=191, tag_tid=14, entry_tid=22423], [object_number=190, tag_tid=14, entry_tid=22422], [object_number=189, tag_tid=14, entry_tid=22421]]], state=connected] 2026-02-22T19:59:44.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:44.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=188, tag_tid=14, entry_tid=22424' 2026-02-22T19:59:44.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=188, tag_tid=14, entry_tid=22424' 2026-02-22T19:59:44.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=22424' '!=' 'object_number=191, tag_tid=14, entry_tid=22219' 2026-02-22T19:59:44.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:44.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=22424' = 'object_number=191, tag_tid=14, entry_tid=22219' 2026-02-22T19:59:44.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=188, tag_tid=14, entry_tid=22424' 2026-02-22T19:59:44.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:44.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:44.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:44.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=22424' 2026-02-22T19:59:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:44.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=22424' 2026-02-22T19:59:44.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:44.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:44.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22424 2026-02-22T19:59:44.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 22424 2026-02-22T19:59:44.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:44.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:44.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:45.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:45.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:45.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:45.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:45.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:45.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:45.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:45.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:45.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:45.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:45.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:45.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:45.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:45.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=22692], [object_number=191, tag_tid=14, entry_tid=22691], [object_number=190, tag_tid=14, entry_tid=22690], [object_number=189, tag_tid=14, entry_tid=22689]]], state=connected] 2026-02-22T19:59:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:45.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:45.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:45.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:45.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:45.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=22692], [object_number=191, tag_tid=14, entry_tid=22691], [object_number=190, tag_tid=14, entry_tid=22690], [object_number=189, tag_tid=14, entry_tid=22689]]], state=connected] 2026-02-22T19:59:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=188, tag_tid=14, entry_tid=22692' 2026-02-22T19:59:45.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=188, tag_tid=14, entry_tid=22692' 2026-02-22T19:59:45.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=22692' '!=' 'object_number=188, tag_tid=14, entry_tid=22424' 2026-02-22T19:59:45.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:45.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=22692' = 'object_number=188, tag_tid=14, entry_tid=22424' 2026-02-22T19:59:45.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=188, tag_tid=14, entry_tid=22692' 2026-02-22T19:59:45.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:45.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:45.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:45.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:45.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=22692' 2026-02-22T19:59:45.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:45.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:45.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:45.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:45.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:45.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:45.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=22692' 2026-02-22T19:59:45.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:45.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:45.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22692 2026-02-22T19:59:45.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 22692 2026-02-22T19:59:45.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:45.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:45.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:45.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:45.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:45.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:45.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:45.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:45.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:45.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:45.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:45.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:45.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:45.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=189, tag_tid=14, entry_tid=22905], [object_number=188, tag_tid=14, entry_tid=22904], [object_number=191, tag_tid=14, entry_tid=22903], [object_number=190, tag_tid=14, entry_tid=22902]]], state=connected] 2026-02-22T19:59:45.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:45.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:45.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:45.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:45.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:45.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:45.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=189, tag_tid=14, entry_tid=22905], [object_number=188, tag_tid=14, entry_tid=22904], [object_number=191, tag_tid=14, entry_tid=22903], [object_number=190, tag_tid=14, entry_tid=22902]]], state=connected] 2026-02-22T19:59:45.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=189, tag_tid=14, entry_tid=22905' 2026-02-22T19:59:45.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=189, tag_tid=14, entry_tid=22905' 2026-02-22T19:59:45.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=189, tag_tid=14, entry_tid=22905' '!=' 'object_number=188, tag_tid=14, entry_tid=22692' 2026-02-22T19:59:45.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:45.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=189, tag_tid=14, entry_tid=22905' = 'object_number=188, tag_tid=14, entry_tid=22692' 2026-02-22T19:59:45.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=189, tag_tid=14, entry_tid=22905' 2026-02-22T19:59:45.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:45.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:45.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:45.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:45.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=189, tag_tid=14, entry_tid=22905' 2026-02-22T19:59:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:45.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:45.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:45.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:45.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=189, tag_tid=14, entry_tid=22905' 2026-02-22T19:59:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22905 2026-02-22T19:59:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 22905 2026-02-22T19:59:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:45.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:45.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:45.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:45.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:45.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:45.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:45.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:45.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:45.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:45.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:45.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:45.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:45.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:45.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:45.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:45.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:45.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:45.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:45.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:45.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:45.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:45.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=191, tag_tid=14, entry_tid=23115], [object_number=190, tag_tid=14, entry_tid=23114], [object_number=189, tag_tid=14, entry_tid=23113], [object_number=188, tag_tid=14, entry_tid=23112]]], state=connected] 2026-02-22T19:59:45.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:46.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:46.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:46.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:46.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:46.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:46.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:46.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:46.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:46.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=191, tag_tid=14, entry_tid=23115], [object_number=190, tag_tid=14, entry_tid=23114], [object_number=189, tag_tid=14, entry_tid=23113], [object_number=188, tag_tid=14, entry_tid=23112]]], state=connected] 2026-02-22T19:59:46.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=191, tag_tid=14, entry_tid=23115' 2026-02-22T19:59:46.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=191, tag_tid=14, entry_tid=23115' 2026-02-22T19:59:46.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=191, tag_tid=14, entry_tid=23115' '!=' 'object_number=189, tag_tid=14, entry_tid=22905' 2026-02-22T19:59:46.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:46.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=191, tag_tid=14, entry_tid=23115' = 'object_number=189, tag_tid=14, entry_tid=22905' 2026-02-22T19:59:46.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=191, tag_tid=14, entry_tid=23115' 2026-02-22T19:59:46.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:46.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:46.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:46.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=191, tag_tid=14, entry_tid=23115' 2026-02-22T19:59:46.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:46.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:46.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:46.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:46.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:46.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=191, tag_tid=14, entry_tid=23115' 2026-02-22T19:59:46.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:46.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23115 2026-02-22T19:59:46.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 23115 2026-02-22T19:59:46.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:46.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:46.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:46.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=191, tag_tid=14, entry_tid=23315], [object_number=190, tag_tid=14, entry_tid=23314], [object_number=189, tag_tid=14, entry_tid=23313], [object_number=188, tag_tid=14, entry_tid=23312]]], state=connected] 2026-02-22T19:59:46.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:46.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:46.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:46.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:46.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=191, tag_tid=14, entry_tid=23315], [object_number=190, tag_tid=14, entry_tid=23314], [object_number=189, tag_tid=14, entry_tid=23313], [object_number=188, tag_tid=14, entry_tid=23312]]], state=connected] 2026-02-22T19:59:46.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=191, tag_tid=14, entry_tid=23315' 2026-02-22T19:59:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=191, tag_tid=14, entry_tid=23315' 2026-02-22T19:59:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=191, tag_tid=14, entry_tid=23315' '!=' 'object_number=191, tag_tid=14, entry_tid=23115' 2026-02-22T19:59:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=191, tag_tid=14, entry_tid=23315' = 'object_number=191, tag_tid=14, entry_tid=23115' 2026-02-22T19:59:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=191, tag_tid=14, entry_tid=23315' 2026-02-22T19:59:46.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:46.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:46.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:46.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=191, tag_tid=14, entry_tid=23315' 2026-02-22T19:59:46.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:46.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:46.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:46.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:46.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:46.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=191, tag_tid=14, entry_tid=23315' 2026-02-22T19:59:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23315 2026-02-22T19:59:46.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 23315 2026-02-22T19:59:46.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:46.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:46.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:46.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:46.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:46.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:46.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:46.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:46.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=23514], [object_number=189, tag_tid=14, entry_tid=23513], [object_number=188, tag_tid=14, entry_tid=23512], [object_number=191, tag_tid=14, entry_tid=23511]]], state=connected] 2026-02-22T19:59:46.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:46.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:46.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:46.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:46.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=23514], [object_number=189, tag_tid=14, entry_tid=23513], [object_number=188, tag_tid=14, entry_tid=23512], [object_number=191, tag_tid=14, entry_tid=23511]]], state=connected] 2026-02-22T19:59:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=190, tag_tid=14, entry_tid=23514' 2026-02-22T19:59:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=190, tag_tid=14, entry_tid=23514' 2026-02-22T19:59:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=23514' '!=' 'object_number=191, tag_tid=14, entry_tid=23315' 2026-02-22T19:59:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=23514' = 'object_number=191, tag_tid=14, entry_tid=23315' 2026-02-22T19:59:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=190, tag_tid=14, entry_tid=23514' 2026-02-22T19:59:46.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:46.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:46.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=23514' 2026-02-22T19:59:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:46.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:46.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:46.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:46.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=23514' 2026-02-22T19:59:46.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23514 2026-02-22T19:59:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 23514 2026-02-22T19:59:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:46.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:46.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:46.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:46.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:46.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:46.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:46.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:46.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:46.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:46.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:46.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:46.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:46.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:46.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:46.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:46.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:46.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:46.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:46.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:47.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:47.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:47.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:47.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:47.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:47.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:47.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:47.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=191, tag_tid=14, entry_tid=23719], [object_number=190, tag_tid=14, entry_tid=23718], [object_number=189, tag_tid=14, entry_tid=23717], [object_number=188, tag_tid=14, entry_tid=23716]]], state=connected] 2026-02-22T19:59:47.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:47.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:47.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:47.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:47.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:47.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:47.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:47.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:47.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:47.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=191, tag_tid=14, entry_tid=23719], [object_number=190, tag_tid=14, entry_tid=23718], [object_number=189, tag_tid=14, entry_tid=23717], [object_number=188, tag_tid=14, entry_tid=23716]]], state=connected] 2026-02-22T19:59:47.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=191, tag_tid=14, entry_tid=23719' 2026-02-22T19:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=191, tag_tid=14, entry_tid=23719' 2026-02-22T19:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=191, tag_tid=14, entry_tid=23719' '!=' 'object_number=190, tag_tid=14, entry_tid=23514' 2026-02-22T19:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=191, tag_tid=14, entry_tid=23719' = 'object_number=190, tag_tid=14, entry_tid=23514' 2026-02-22T19:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=191, tag_tid=14, entry_tid=23719' 2026-02-22T19:59:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:47.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=191, tag_tid=14, entry_tid=23719' 2026-02-22T19:59:47.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:47.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:47.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:47.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:47.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:47.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=191, tag_tid=14, entry_tid=23719' 2026-02-22T19:59:47.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:47.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:47.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23719 2026-02-22T19:59:47.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 23719 2026-02-22T19:59:47.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:47.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:47.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:47.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:47.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:47.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:47.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:47.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:47.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:47.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:47.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:47.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:47.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:47.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:47.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:47.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:47.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:47.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:47.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:47.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:47.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:47.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:47.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:47.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:47.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:47.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:47.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:47.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:47.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:47.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=24040], [object_number=191, tag_tid=14, entry_tid=24039], [object_number=190, tag_tid=14, entry_tid=24038], [object_number=189, tag_tid=14, entry_tid=24037]]], state=connected] 2026-02-22T19:59:47.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:47.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:47.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:47.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:47.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:47.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:47.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:47.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:47.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:47.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:47.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=24040], [object_number=191, tag_tid=14, entry_tid=24039], [object_number=190, tag_tid=14, entry_tid=24038], [object_number=189, tag_tid=14, entry_tid=24037]]], state=connected] 2026-02-22T19:59:47.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=188, tag_tid=14, entry_tid=24040' 2026-02-22T19:59:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=188, tag_tid=14, entry_tid=24040' 2026-02-22T19:59:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=24040' '!=' 'object_number=191, tag_tid=14, entry_tid=23719' 2026-02-22T19:59:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=24040' = 'object_number=191, tag_tid=14, entry_tid=23719' 2026-02-22T19:59:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=188, tag_tid=14, entry_tid=24040' 2026-02-22T19:59:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=24040' 2026-02-22T19:59:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:47.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=24040' 2026-02-22T19:59:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:47.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24040 2026-02-22T19:59:47.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 24040 2026-02-22T19:59:47.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:47.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:47.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:47.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:47.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:47.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:47.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:47.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:47.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:47.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:47.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:47.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:47.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:47.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:47.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:47.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:47.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:47.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:47.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:47.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:47.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:48.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:48.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:48.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:48.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=24242], [object_number=189, tag_tid=14, entry_tid=24241], [object_number=188, tag_tid=14, entry_tid=24240], [object_number=191, tag_tid=14, entry_tid=24239]]], state=connected] 2026-02-22T19:59:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:48.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:48.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:48.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:48.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:48.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:48.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:48.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:48.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=24242], [object_number=189, tag_tid=14, entry_tid=24241], [object_number=188, tag_tid=14, entry_tid=24240], [object_number=191, tag_tid=14, entry_tid=24239]]], state=connected] 2026-02-22T19:59:48.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=190, tag_tid=14, entry_tid=24242' 2026-02-22T19:59:48.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=190, tag_tid=14, entry_tid=24242' 2026-02-22T19:59:48.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=24242' '!=' 'object_number=188, tag_tid=14, entry_tid=24040' 2026-02-22T19:59:48.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:48.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=24242' = 'object_number=188, tag_tid=14, entry_tid=24040' 2026-02-22T19:59:48.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=190, tag_tid=14, entry_tid=24242' 2026-02-22T19:59:48.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:48.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:48.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:48.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:48.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=24242' 2026-02-22T19:59:48.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:48.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:48.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:48.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:48.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:48.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:48.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=24242' 2026-02-22T19:59:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:48.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24242 2026-02-22T19:59:48.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 24242 2026-02-22T19:59:48.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:48.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:48.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:48.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:48.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:48.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:48.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:48.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:48.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:48.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:48.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:48.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:48.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:48.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:48.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:48.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:48.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:48.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=24448], [object_number=191, tag_tid=14, entry_tid=24447], [object_number=190, tag_tid=14, entry_tid=24446], [object_number=189, tag_tid=14, entry_tid=24445]]], state=connected] 2026-02-22T19:59:48.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:48.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:48.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:48.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:48.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:48.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:48.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=24448], [object_number=191, tag_tid=14, entry_tid=24447], [object_number=190, tag_tid=14, entry_tid=24446], [object_number=189, tag_tid=14, entry_tid=24445]]], state=connected] 2026-02-22T19:59:48.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=188, tag_tid=14, entry_tid=24448' 2026-02-22T19:59:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=188, tag_tid=14, entry_tid=24448' 2026-02-22T19:59:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=24448' '!=' 'object_number=190, tag_tid=14, entry_tid=24242' 2026-02-22T19:59:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=24448' = 'object_number=190, tag_tid=14, entry_tid=24242' 2026-02-22T19:59:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=188, tag_tid=14, entry_tid=24448' 2026-02-22T19:59:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:48.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:48.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=24448' 2026-02-22T19:59:48.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:48.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:48.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:48.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=24448' 2026-02-22T19:59:48.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:48.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:48.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24448 2026-02-22T19:59:48.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 24448 2026-02-22T19:59:48.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:48.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:48.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:48.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:48.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:48.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:48.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:48.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:48.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:48.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:48.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:48.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:48.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:48.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=24786], [object_number=189, tag_tid=14, entry_tid=24785], [object_number=188, tag_tid=14, entry_tid=24784], [object_number=191, tag_tid=14, entry_tid=24783]]], state=connected] 2026-02-22T19:59:48.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:49.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:49.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:49.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:49.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=24786], [object_number=189, tag_tid=14, entry_tid=24785], [object_number=188, tag_tid=14, entry_tid=24784], [object_number=191, tag_tid=14, entry_tid=24783]]], state=connected] 2026-02-22T19:59:49.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=190, tag_tid=14, entry_tid=24786' 2026-02-22T19:59:49.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=190, tag_tid=14, entry_tid=24786' 2026-02-22T19:59:49.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=24786' '!=' 'object_number=188, tag_tid=14, entry_tid=24448' 2026-02-22T19:59:49.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:49.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=24786' = 'object_number=188, tag_tid=14, entry_tid=24448' 2026-02-22T19:59:49.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=190, tag_tid=14, entry_tid=24786' 2026-02-22T19:59:49.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:49.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:49.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=24786' 2026-02-22T19:59:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:49.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=24786' 2026-02-22T19:59:49.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:49.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24786 2026-02-22T19:59:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 24786 2026-02-22T19:59:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:49.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:49.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:49.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:49.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:49.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:49.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:49.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:49.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:49.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:49.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=25004], [object_number=191, tag_tid=14, entry_tid=25003], [object_number=190, tag_tid=14, entry_tid=25002], [object_number=189, tag_tid=14, entry_tid=25001]]], state=connected] 2026-02-22T19:59:49.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:49.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:49.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:49.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:49.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:49.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:49.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:49.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:49.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:49.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:49.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:49.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=25004], [object_number=191, tag_tid=14, entry_tid=25003], [object_number=190, tag_tid=14, entry_tid=25002], [object_number=189, tag_tid=14, entry_tid=25001]]], state=connected] 2026-02-22T19:59:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=188, tag_tid=14, entry_tid=25004' 2026-02-22T19:59:49.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=188, tag_tid=14, entry_tid=25004' 2026-02-22T19:59:49.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=25004' '!=' 'object_number=190, tag_tid=14, entry_tid=24786' 2026-02-22T19:59:49.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:49.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=25004' = 'object_number=190, tag_tid=14, entry_tid=24786' 2026-02-22T19:59:49.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=188, tag_tid=14, entry_tid=25004' 2026-02-22T19:59:49.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:49.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:49.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:49.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=25004' 2026-02-22T19:59:49.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:49.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:49.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:49.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=25004' 2026-02-22T19:59:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25004 2026-02-22T19:59:49.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 25004 2026-02-22T19:59:49.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:49.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:49.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:49.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:49.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:49.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:49.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:49.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:49.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:49.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:49.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:49.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:49.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:49.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:49.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:49.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:49.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:49.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:49.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:49.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:49.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=25222], [object_number=189, tag_tid=14, entry_tid=25221], [object_number=188, tag_tid=14, entry_tid=25220], [object_number=191, tag_tid=14, entry_tid=25219]]], state=connected] 2026-02-22T19:59:49.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:49.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:49.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:49.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:49.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:49.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:49.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:49.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:49.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:49.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:49.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:49.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:49.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:49.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:49.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=190, tag_tid=14, entry_tid=25222], [object_number=189, tag_tid=14, entry_tid=25221], [object_number=188, tag_tid=14, entry_tid=25220], [object_number=191, tag_tid=14, entry_tid=25219]]], state=connected] 2026-02-22T19:59:49.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:49.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=190, tag_tid=14, entry_tid=25222' 2026-02-22T19:59:49.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=190, tag_tid=14, entry_tid=25222' 2026-02-22T19:59:49.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=25222' '!=' 'object_number=188, tag_tid=14, entry_tid=25004' 2026-02-22T19:59:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=190, tag_tid=14, entry_tid=25222' = 'object_number=188, tag_tid=14, entry_tid=25004' 2026-02-22T19:59:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=190, tag_tid=14, entry_tid=25222' 2026-02-22T19:59:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=25222' 2026-02-22T19:59:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:49.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:49.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:49.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:49.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=190, tag_tid=14, entry_tid=25222' 2026-02-22T19:59:49.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:49.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:49.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25222 2026-02-22T19:59:49.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 25222 2026-02-22T19:59:49.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:49.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:49.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:49.885 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5898629405012392) > weight (0.25) 2026-02-22T19:59:49.885 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7441148198662124) > weight (0.25) 2026-02-22T19:59:49.885 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2026-02-22T19:59:49.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2026-02-22T19:59:49.886 INFO:tasks.rbd_mirror_thrash:waiting for 5.503726119526011 secs before reviving daemons 2026-02-22T19:59:49.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T19:59:49.874+0000 7f02f0f03640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 38832) UID: 1000 2026-02-22T19:59:49.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:49.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:49.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:49.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:49.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:49.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:49.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:49.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:49.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:49.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:49.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:49.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:49.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:50.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:50.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:50.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:50.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:50.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:50.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:50.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:50.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:50.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:50.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=25436], [object_number=191, tag_tid=14, entry_tid=25435], [object_number=190, tag_tid=14, entry_tid=25434], [object_number=189, tag_tid=14, entry_tid=25433]]], state=connected] 2026-02-22T19:59:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=25436], [object_number=191, tag_tid=14, entry_tid=25435], [object_number=190, tag_tid=14, entry_tid=25434], [object_number=189, tag_tid=14, entry_tid=25433]]], state=connected] 2026-02-22T19:59:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=188, tag_tid=14, entry_tid=25436' 2026-02-22T19:59:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=188, tag_tid=14, entry_tid=25436' 2026-02-22T19:59:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=25436' '!=' 'object_number=190, tag_tid=14, entry_tid=25222' 2026-02-22T19:59:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:50.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=25436' = 'object_number=190, tag_tid=14, entry_tid=25222' 2026-02-22T19:59:50.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=188, tag_tid=14, entry_tid=25436' 2026-02-22T19:59:50.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:50.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:50.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=25436' 2026-02-22T19:59:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:50.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:50.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=25436' 2026-02-22T19:59:50.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:50.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:50.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25436 2026-02-22T19:59:50.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 25436 2026-02-22T19:59:50.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:50.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:50.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:50.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:50.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:50.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:50.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:50.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:50.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:50.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:50.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:50.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:50.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:50.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:50.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:50.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:50.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:50.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:50.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:50.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:50.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:50.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:50.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:50.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:50.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:50.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:50.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:50.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:50.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=25648], [object_number=191, tag_tid=14, entry_tid=25647], [object_number=190, tag_tid=14, entry_tid=25646], [object_number=189, tag_tid=14, entry_tid=25645]]], state=connected] 2026-02-22T19:59:50.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:50.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:50.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:50.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:50.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:50.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:50.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:50.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:50.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 47 2026-02-22T19:59:50.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:50.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:50.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:50.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=188, tag_tid=14, entry_tid=25648], [object_number=191, tag_tid=14, entry_tid=25647], [object_number=190, tag_tid=14, entry_tid=25646], [object_number=189, tag_tid=14, entry_tid=25645]]], state=connected] 2026-02-22T19:59:50.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=188, tag_tid=14, entry_tid=25648' 2026-02-22T19:59:50.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=188, tag_tid=14, entry_tid=25648' 2026-02-22T19:59:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=25648' '!=' 'object_number=188, tag_tid=14, entry_tid=25436' 2026-02-22T19:59:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=188, tag_tid=14, entry_tid=25648' = 'object_number=188, tag_tid=14, entry_tid=25436' 2026-02-22T19:59:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=188, tag_tid=14, entry_tid=25648' 2026-02-22T19:59:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:50.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:50.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=25648' 2026-02-22T19:59:50.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:50.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:50.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:50.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:50.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:50.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:50.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:50.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=188, tag_tid=14, entry_tid=25648' 2026-02-22T19:59:50.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:50.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:50.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25648 2026-02-22T19:59:50.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 25648 2026-02-22T19:59:50.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:50.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:50.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:50.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:50.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:50.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:50.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:50.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:50.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:50.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:50.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:50.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:50.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:50.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:50.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:50.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:50.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:50.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:50.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:50.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:50.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:50.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:50.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:50.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:50.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:50.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=25920], [object_number=195, tag_tid=14, entry_tid=25919], [object_number=194, tag_tid=14, entry_tid=25918], [object_number=193, tag_tid=14, entry_tid=25917]]], state=connected] 2026-02-22T19:59:50.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:51.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:51.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:51.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:51.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:51.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=25920], [object_number=195, tag_tid=14, entry_tid=25919], [object_number=194, tag_tid=14, entry_tid=25918], [object_number=193, tag_tid=14, entry_tid=25917]]], state=connected] 2026-02-22T19:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=192, tag_tid=14, entry_tid=25920' 2026-02-22T19:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=192, tag_tid=14, entry_tid=25920' 2026-02-22T19:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=25920' '!=' 'object_number=188, tag_tid=14, entry_tid=25648' 2026-02-22T19:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=25920' = 'object_number=188, tag_tid=14, entry_tid=25648' 2026-02-22T19:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=192, tag_tid=14, entry_tid=25920' 2026-02-22T19:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:51.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=25920' 2026-02-22T19:59:51.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:51.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:51.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:51.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=25920' 2026-02-22T19:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25920 2026-02-22T19:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 25920 2026-02-22T19:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:51.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:51.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:51.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:51.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:51.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:51.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:51.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:51.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:51.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:51.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:51.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:51.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:51.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:51.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:51.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:51.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:51.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:51.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:51.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:51.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:51.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:51.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:51.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:51.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:51.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:51.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:51.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:51.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:51.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:51.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:51.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:51.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:51.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:51.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:51.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:51.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:51.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:51.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:51.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:51.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=26148], [object_number=195, tag_tid=14, entry_tid=26147], [object_number=194, tag_tid=14, entry_tid=26146], [object_number=193, tag_tid=14, entry_tid=26145]]], state=connected] 2026-02-22T19:59:51.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:51.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:51.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:51.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:51.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:51.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=26148], [object_number=195, tag_tid=14, entry_tid=26147], [object_number=194, tag_tid=14, entry_tid=26146], [object_number=193, tag_tid=14, entry_tid=26145]]], state=connected] 2026-02-22T19:59:51.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=192, tag_tid=14, entry_tid=26148' 2026-02-22T19:59:51.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=192, tag_tid=14, entry_tid=26148' 2026-02-22T19:59:51.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=26148' '!=' 'object_number=192, tag_tid=14, entry_tid=25920' 2026-02-22T19:59:51.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:51.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=26148' = 'object_number=192, tag_tid=14, entry_tid=25920' 2026-02-22T19:59:51.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=192, tag_tid=14, entry_tid=26148' 2026-02-22T19:59:51.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:51.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:51.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:51.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=26148' 2026-02-22T19:59:51.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:51.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:51.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:51.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:51.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:51.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=26148' 2026-02-22T19:59:51.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:51.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26148 2026-02-22T19:59:51.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 26148 2026-02-22T19:59:51.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:51.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:51.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:51.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:51.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:51.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:51.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:51.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:51.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:51.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:51.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:51.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:51.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:51.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:51.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:51.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:51.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:51.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:51.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:51.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:51.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:51.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:51.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:51.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:51.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:51.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:51.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:51.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:51.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=195, tag_tid=14, entry_tid=26351], [object_number=194, tag_tid=14, entry_tid=26350], [object_number=193, tag_tid=14, entry_tid=26349], [object_number=192, tag_tid=14, entry_tid=26348]]], state=connected] 2026-02-22T19:59:51.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:51.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:51.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:51.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:51.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:51.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:51.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:51.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:51.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:51.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:51.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=195, tag_tid=14, entry_tid=26351], [object_number=194, tag_tid=14, entry_tid=26350], [object_number=193, tag_tid=14, entry_tid=26349], [object_number=192, tag_tid=14, entry_tid=26348]]], state=connected] 2026-02-22T19:59:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:51.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=195, tag_tid=14, entry_tid=26351' 2026-02-22T19:59:51.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=195, tag_tid=14, entry_tid=26351' 2026-02-22T19:59:51.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=195, tag_tid=14, entry_tid=26351' '!=' 'object_number=192, tag_tid=14, entry_tid=26148' 2026-02-22T19:59:51.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:51.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=195, tag_tid=14, entry_tid=26351' = 'object_number=192, tag_tid=14, entry_tid=26148' 2026-02-22T19:59:51.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=195, tag_tid=14, entry_tid=26351' 2026-02-22T19:59:51.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:51.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:51.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:51.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=195, tag_tid=14, entry_tid=26351' 2026-02-22T19:59:51.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:51.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:51.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:51.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:51.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:51.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=195, tag_tid=14, entry_tid=26351' 2026-02-22T19:59:51.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:51.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26351 2026-02-22T19:59:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 26351 2026-02-22T19:59:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:51.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:51.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:51.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:51.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:51.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:51.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:52.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:52.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:52.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:52.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:52.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:52.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:52.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:52.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:52.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=193, tag_tid=14, entry_tid=26557], [object_number=192, tag_tid=14, entry_tid=26556], [object_number=195, tag_tid=14, entry_tid=26555], [object_number=194, tag_tid=14, entry_tid=26554]]], state=connected] 2026-02-22T19:59:52.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:52.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:52.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:52.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:52.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:52.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:52.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:52.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:52.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=193, tag_tid=14, entry_tid=26557], [object_number=192, tag_tid=14, entry_tid=26556], [object_number=195, tag_tid=14, entry_tid=26555], [object_number=194, tag_tid=14, entry_tid=26554]]], state=connected] 2026-02-22T19:59:52.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=193, tag_tid=14, entry_tid=26557' 2026-02-22T19:59:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=193, tag_tid=14, entry_tid=26557' 2026-02-22T19:59:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=193, tag_tid=14, entry_tid=26557' '!=' 'object_number=195, tag_tid=14, entry_tid=26351' 2026-02-22T19:59:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=193, tag_tid=14, entry_tid=26557' = 'object_number=195, tag_tid=14, entry_tid=26351' 2026-02-22T19:59:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=193, tag_tid=14, entry_tid=26557' 2026-02-22T19:59:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=14, entry_tid=26557' 2026-02-22T19:59:52.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:52.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:52.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:52.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:52.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:52.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=14, entry_tid=26557' 2026-02-22T19:59:52.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:52.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26557 2026-02-22T19:59:52.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 26557 2026-02-22T19:59:52.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:52.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:52.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:52.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:52.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:52.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:52.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:52.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:52.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:52.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:52.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:52.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:52.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:52.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:52.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:52.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:52.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:52.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:52.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:52.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:52.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:52.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:52.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:52.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:52.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:52.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:52.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=26760], [object_number=195, tag_tid=14, entry_tid=26759], [object_number=194, tag_tid=14, entry_tid=26758], [object_number=193, tag_tid=14, entry_tid=26757]]], state=connected] 2026-02-22T19:59:52.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:52.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:52.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:52.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:52.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=26760], [object_number=195, tag_tid=14, entry_tid=26759], [object_number=194, tag_tid=14, entry_tid=26758], [object_number=193, tag_tid=14, entry_tid=26757]]], state=connected] 2026-02-22T19:59:52.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=192, tag_tid=14, entry_tid=26760' 2026-02-22T19:59:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=192, tag_tid=14, entry_tid=26760' 2026-02-22T19:59:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=26760' '!=' 'object_number=193, tag_tid=14, entry_tid=26557' 2026-02-22T19:59:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=26760' = 'object_number=193, tag_tid=14, entry_tid=26557' 2026-02-22T19:59:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=192, tag_tid=14, entry_tid=26760' 2026-02-22T19:59:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=26760' 2026-02-22T19:59:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:52.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:52.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=26760' 2026-02-22T19:59:52.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:52.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26760 2026-02-22T19:59:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 26760 2026-02-22T19:59:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:52.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:52.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:52.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:52.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:52.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:52.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:52.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:52.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:52.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:52.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:52.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:52.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:52.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:52.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:52.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:52.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=193, tag_tid=14, entry_tid=26981], [object_number=192, tag_tid=14, entry_tid=26980], [object_number=195, tag_tid=14, entry_tid=26979], [object_number=194, tag_tid=14, entry_tid=26978]]], state=connected] 2026-02-22T19:59:52.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:52.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:52.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:52.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:52.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:52.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:52.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=193, tag_tid=14, entry_tid=26981], [object_number=192, tag_tid=14, entry_tid=26980], [object_number=195, tag_tid=14, entry_tid=26979], [object_number=194, tag_tid=14, entry_tid=26978]]], state=connected] 2026-02-22T19:59:52.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:52.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=193, tag_tid=14, entry_tid=26981' 2026-02-22T19:59:52.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=193, tag_tid=14, entry_tid=26981' 2026-02-22T19:59:52.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=193, tag_tid=14, entry_tid=26981' '!=' 'object_number=192, tag_tid=14, entry_tid=26760' 2026-02-22T19:59:52.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:52.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=193, tag_tid=14, entry_tid=26981' = 'object_number=192, tag_tid=14, entry_tid=26760' 2026-02-22T19:59:52.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=193, tag_tid=14, entry_tid=26981' 2026-02-22T19:59:52.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:52.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:52.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:52.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=14, entry_tid=26981' 2026-02-22T19:59:52.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:52.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:52.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:52.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:52.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:52.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=14, entry_tid=26981' 2026-02-22T19:59:52.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:52.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26981 2026-02-22T19:59:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 26981 2026-02-22T19:59:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:53.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:53.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:53.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:53.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:53.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:53.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:53.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:53.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:53.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:53.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:53.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:53.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:53.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:53.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=27216], [object_number=195, tag_tid=14, entry_tid=27215], [object_number=194, tag_tid=14, entry_tid=27214], [object_number=193, tag_tid=14, entry_tid=27213]]], state=connected] 2026-02-22T19:59:53.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:53.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:53.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:53.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:53.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=27216], [object_number=195, tag_tid=14, entry_tid=27215], [object_number=194, tag_tid=14, entry_tid=27214], [object_number=193, tag_tid=14, entry_tid=27213]]], state=connected] 2026-02-22T19:59:53.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=192, tag_tid=14, entry_tid=27216' 2026-02-22T19:59:53.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=192, tag_tid=14, entry_tid=27216' 2026-02-22T19:59:53.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=27216' '!=' 'object_number=193, tag_tid=14, entry_tid=26981' 2026-02-22T19:59:53.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:53.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=27216' = 'object_number=193, tag_tid=14, entry_tid=26981' 2026-02-22T19:59:53.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=192, tag_tid=14, entry_tid=27216' 2026-02-22T19:59:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=27216' 2026-02-22T19:59:53.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:53.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:53.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:53.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:53.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=27216' 2026-02-22T19:59:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27216 2026-02-22T19:59:53.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 27216 2026-02-22T19:59:53.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:53.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:53.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:53.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:53.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:53.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:53.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:53.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:53.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:53.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:53.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:53.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:53.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:53.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:53.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:53.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:53.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:53.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:53.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:53.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=194, tag_tid=14, entry_tid=27422], [object_number=193, tag_tid=14, entry_tid=27421], [object_number=192, tag_tid=14, entry_tid=27420], [object_number=195, tag_tid=14, entry_tid=27419]]], state=connected] 2026-02-22T19:59:53.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:53.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:53.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:53.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=194, tag_tid=14, entry_tid=27422], [object_number=193, tag_tid=14, entry_tid=27421], [object_number=192, tag_tid=14, entry_tid=27420], [object_number=195, tag_tid=14, entry_tid=27419]]], state=connected] 2026-02-22T19:59:53.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=194, tag_tid=14, entry_tid=27422' 2026-02-22T19:59:53.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=194, tag_tid=14, entry_tid=27422' 2026-02-22T19:59:53.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=194, tag_tid=14, entry_tid=27422' '!=' 'object_number=192, tag_tid=14, entry_tid=27216' 2026-02-22T19:59:53.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:53.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=194, tag_tid=14, entry_tid=27422' = 'object_number=192, tag_tid=14, entry_tid=27216' 2026-02-22T19:59:53.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=194, tag_tid=14, entry_tid=27422' 2026-02-22T19:59:53.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:53.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:53.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=194, tag_tid=14, entry_tid=27422' 2026-02-22T19:59:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:53.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=194, tag_tid=14, entry_tid=27422' 2026-02-22T19:59:53.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:53.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27422 2026-02-22T19:59:53.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 27422 2026-02-22T19:59:53.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:53.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:53.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:53.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:53.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:53.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:53.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:53.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:53.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:53.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:53.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:53.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:53.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:53.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:53.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:53.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:53.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:53.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:53.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:53.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:53.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:53.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:53.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:53.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:53.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:53.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:53.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:53.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:53.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:53.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:53.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:53.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:53.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:53.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:53.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:53.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:53.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:53.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=194, tag_tid=14, entry_tid=27638], [object_number=193, tag_tid=14, entry_tid=27637], [object_number=192, tag_tid=14, entry_tid=27636], [object_number=195, tag_tid=14, entry_tid=27635]]], state=connected] 2026-02-22T19:59:53.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:53.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:53.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:53.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:53.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:53.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:53.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:53.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:53.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:53.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:53.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:53.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:53.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:53.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=194, tag_tid=14, entry_tid=27638], [object_number=193, tag_tid=14, entry_tid=27637], [object_number=192, tag_tid=14, entry_tid=27636], [object_number=195, tag_tid=14, entry_tid=27635]]], state=connected] 2026-02-22T19:59:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=194, tag_tid=14, entry_tid=27638' 2026-02-22T19:59:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=194, tag_tid=14, entry_tid=27638' 2026-02-22T19:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=194, tag_tid=14, entry_tid=27638' '!=' 'object_number=194, tag_tid=14, entry_tid=27422' 2026-02-22T19:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=194, tag_tid=14, entry_tid=27638' = 'object_number=194, tag_tid=14, entry_tid=27422' 2026-02-22T19:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=194, tag_tid=14, entry_tid=27638' 2026-02-22T19:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:53.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:53.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:53.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=194, tag_tid=14, entry_tid=27638' 2026-02-22T19:59:53.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:53.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:53.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:53.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:53.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:53.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=194, tag_tid=14, entry_tid=27638' 2026-02-22T19:59:53.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:53.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:53.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27638 2026-02-22T19:59:53.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 27638 2026-02-22T19:59:53.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:53.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:53.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:54.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:54.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:54.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:54.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:54.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:54.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:54.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:54.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:54.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:54.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:54.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:54.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:54.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:54.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:54.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:54.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:54.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:54.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:54.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:54.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:54.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:54.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:54.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:54.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:54.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:54.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:54.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:54.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:54.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=193, tag_tid=14, entry_tid=27837], [object_number=192, tag_tid=14, entry_tid=27836], [object_number=195, tag_tid=14, entry_tid=27835], [object_number=194, tag_tid=14, entry_tid=27834]]], state=connected] 2026-02-22T19:59:54.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:54.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:54.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:54.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:54.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=193, tag_tid=14, entry_tid=27837], [object_number=192, tag_tid=14, entry_tid=27836], [object_number=195, tag_tid=14, entry_tid=27835], [object_number=194, tag_tid=14, entry_tid=27834]]], state=connected] 2026-02-22T19:59:54.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=193, tag_tid=14, entry_tid=27837' 2026-02-22T19:59:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=193, tag_tid=14, entry_tid=27837' 2026-02-22T19:59:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=193, tag_tid=14, entry_tid=27837' '!=' 'object_number=194, tag_tid=14, entry_tid=27638' 2026-02-22T19:59:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=193, tag_tid=14, entry_tid=27837' = 'object_number=194, tag_tid=14, entry_tid=27638' 2026-02-22T19:59:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=193, tag_tid=14, entry_tid=27837' 2026-02-22T19:59:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=14, entry_tid=27837' 2026-02-22T19:59:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:54.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:54.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:54.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:54.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:54.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=14, entry_tid=27837' 2026-02-22T19:59:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27837 2026-02-22T19:59:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 27837 2026-02-22T19:59:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:54.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:54.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:54.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:54.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:54.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:54.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:54.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:54.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:54.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:54.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:54.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:54.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:54.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:54.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:54.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:54.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:54.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:54.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:54.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:54.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:54.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:54.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=195, tag_tid=14, entry_tid=28031], [object_number=194, tag_tid=14, entry_tid=28030], [object_number=193, tag_tid=14, entry_tid=28029], [object_number=192, tag_tid=14, entry_tid=28028]]], state=connected] 2026-02-22T19:59:54.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:54.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:54.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:54.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:54.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:54.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:54.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:54.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:54.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:54.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:54.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=195, tag_tid=14, entry_tid=28031], [object_number=194, tag_tid=14, entry_tid=28030], [object_number=193, tag_tid=14, entry_tid=28029], [object_number=192, tag_tid=14, entry_tid=28028]]], state=connected] 2026-02-22T19:59:54.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=195, tag_tid=14, entry_tid=28031' 2026-02-22T19:59:54.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=195, tag_tid=14, entry_tid=28031' 2026-02-22T19:59:54.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=195, tag_tid=14, entry_tid=28031' '!=' 'object_number=193, tag_tid=14, entry_tid=27837' 2026-02-22T19:59:54.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:54.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=195, tag_tid=14, entry_tid=28031' = 'object_number=193, tag_tid=14, entry_tid=27837' 2026-02-22T19:59:54.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=195, tag_tid=14, entry_tid=28031' 2026-02-22T19:59:54.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:54.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:54.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:54.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:54.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=195, tag_tid=14, entry_tid=28031' 2026-02-22T19:59:54.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:54.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:54.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:54.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=195, tag_tid=14, entry_tid=28031' 2026-02-22T19:59:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:54.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28031 2026-02-22T19:59:54.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 28031 2026-02-22T19:59:54.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:54.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:54.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:54.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:54.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:54.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:54.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:54.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:54.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:54.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:54.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:54.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:54.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:55.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:55.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:55.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:55.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=194, tag_tid=14, entry_tid=28226], [object_number=193, tag_tid=14, entry_tid=28225], [object_number=192, tag_tid=14, entry_tid=28224], [object_number=195, tag_tid=14, entry_tid=28223]]], state=connected] 2026-02-22T19:59:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:55.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:55.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:55.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:55.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:55.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:55.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:55.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:55.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:55.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:55.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:55.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:55.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=194, tag_tid=14, entry_tid=28226], [object_number=193, tag_tid=14, entry_tid=28225], [object_number=192, tag_tid=14, entry_tid=28224], [object_number=195, tag_tid=14, entry_tid=28223]]], state=connected] 2026-02-22T19:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=194, tag_tid=14, entry_tid=28226' 2026-02-22T19:59:55.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=194, tag_tid=14, entry_tid=28226' 2026-02-22T19:59:55.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=194, tag_tid=14, entry_tid=28226' '!=' 'object_number=195, tag_tid=14, entry_tid=28031' 2026-02-22T19:59:55.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:55.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=194, tag_tid=14, entry_tid=28226' = 'object_number=195, tag_tid=14, entry_tid=28031' 2026-02-22T19:59:55.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=194, tag_tid=14, entry_tid=28226' 2026-02-22T19:59:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:55.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=194, tag_tid=14, entry_tid=28226' 2026-02-22T19:59:55.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:55.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=194, tag_tid=14, entry_tid=28226' 2026-02-22T19:59:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28226 2026-02-22T19:59:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 28226 2026-02-22T19:59:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:55.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:55.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:55.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:55.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:55.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:55.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:55.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:55.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:55.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:55.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=194, tag_tid=14, entry_tid=28422], [object_number=193, tag_tid=14, entry_tid=28421], [object_number=192, tag_tid=14, entry_tid=28420], [object_number=195, tag_tid=14, entry_tid=28419]]], state=connected] 2026-02-22T19:59:55.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:55.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:55.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:55.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:55.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:55.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:55.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:55.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:55.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.390 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2026-02-22T19:59:55.391 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T19:59:55.391 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2026-02-22T19:59:55.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2026-02-22T19:59:55.391 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2026-02-22T19:59:55.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2026-02-22T19:59:55.393 INFO:tasks.rbd_mirror_thrash:waiting for 88.63413775611194 secs before thrashing 2026-02-22T19:59:55.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:55.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:55.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:55.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:55.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=194, tag_tid=14, entry_tid=28422], [object_number=193, tag_tid=14, entry_tid=28421], [object_number=192, tag_tid=14, entry_tid=28420], [object_number=195, tag_tid=14, entry_tid=28419]]], state=connected] 2026-02-22T19:59:55.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=194, tag_tid=14, entry_tid=28422' 2026-02-22T19:59:55.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=194, tag_tid=14, entry_tid=28422' 2026-02-22T19:59:55.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=194, tag_tid=14, entry_tid=28422' '!=' 'object_number=194, tag_tid=14, entry_tid=28226' 2026-02-22T19:59:55.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:55.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=194, tag_tid=14, entry_tid=28422' = 'object_number=194, tag_tid=14, entry_tid=28226' 2026-02-22T19:59:55.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=194, tag_tid=14, entry_tid=28422' 2026-02-22T19:59:55.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:55.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:55.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=194, tag_tid=14, entry_tid=28422' 2026-02-22T19:59:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:55.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=194, tag_tid=14, entry_tid=28422' 2026-02-22T19:59:55.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:55.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28422 2026-02-22T19:59:55.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 28422 2026-02-22T19:59:55.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:55.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:55.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:55.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:55.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:55.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:55.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:55.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:55.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:55.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:55.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:55.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:55.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:55.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=28468], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466], [object_number=193, tag_tid=14, entry_tid=28465]]], state=connected] 2026-02-22T19:59:55.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:55.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:55.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:55.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:55.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:55.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:55.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:55.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:55.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:55.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:55.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=28468], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466], [object_number=193, tag_tid=14, entry_tid=28465]]], state=connected] 2026-02-22T19:59:55.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:55.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=192, tag_tid=14, entry_tid=28468' 2026-02-22T19:59:55.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=192, tag_tid=14, entry_tid=28468' 2026-02-22T19:59:55.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=28468' '!=' 'object_number=194, tag_tid=14, entry_tid=28422' 2026-02-22T19:59:55.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T19:59:55.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=28468' = 'object_number=194, tag_tid=14, entry_tid=28422' 2026-02-22T19:59:55.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=192, tag_tid=14, entry_tid=28468' 2026-02-22T19:59:55.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:55.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:55.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=15 2026-02-22T19:59:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=28468' 2026-02-22T19:59:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T19:59:55.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=14 2026-02-22T19:59:55.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:55.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:55.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T19:59:55.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=192, tag_tid=14, entry_tid=28468' 2026-02-22T19:59:55.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T19:59:55.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T19:59:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28468 2026-02-22T19:59:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 15 = 14 -a 1 -le 28468 2026-02-22T19:59:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T19:59:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T19:59:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:56.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:56.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:56.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:56.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:56.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:56.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:56.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:56.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:56.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:56.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:56.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:56.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:56.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:56.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=28468], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466], [object_number=193, tag_tid=14, entry_tid=28465]]], state=connected] 2026-02-22T19:59:56.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:56.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:56.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:56.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:56.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:56.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:56.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:56.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:56.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:56.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:56.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:56.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:56.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:56.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:56.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:56.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:56.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:56.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:56.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=192, tag_tid=14, entry_tid=28468], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466], [object_number=193, tag_tid=14, entry_tid=28465]]], state=connected] 2026-02-22T19:59:56.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=192, tag_tid=14, entry_tid=28468' 2026-02-22T19:59:56.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=192, tag_tid=14, entry_tid=28468' 2026-02-22T19:59:56.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=192, tag_tid=14, entry_tid=28468' '!=' 'object_number=192, tag_tid=14, entry_tid=28468' 2026-02-22T19:59:56.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:56.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T19:59:56.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T19:59:56.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T19:59:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T19:59:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T19:59:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:56.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T19:59:56.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:59:56.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T19:59:56.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T19:59:56.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T19:59:56.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:56.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T19:59:56.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T19:59:56.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T19:59:56.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T19:59:56.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T19:59:56.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:56.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:56.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:56.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:56.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:56.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:56.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:56.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:56.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:56.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:56.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T19:59:56.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:56.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:56.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:56.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T19:59:56.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T19:59:56.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T19:59:56.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T19:59:56.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T19:59:56.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:56.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T19:59:56.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:56.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T19:59:56.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 48 2026-02-22T19:59:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 48 2026-02-22T19:59:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T19:59:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=193, tag_tid=15, entry_tid=1], [object_number=192, tag_tid=15, entry_tid=0], [object_number=195, tag_tid=14, entry_tid=28467], [object_number=194, tag_tid=14, entry_tid=28466]]], state=connected] 2026-02-22T19:59:56.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T19:59:56.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:56.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=193, tag_tid=15, entry_tid=1' -a 'object_number=193, tag_tid=15, entry_tid=1' = 'object_number=193, tag_tid=15, entry_tid=1' 2026-02-22T19:59:56.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:56.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2026-02-22T19:59:56.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T19:59:56.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:56.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:56.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap5 2026-02-22T19:59:56.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T19:59:56.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T19:59:56.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T19:59:57.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2026-02-22T19:59:57.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T19:59:57.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T19:59:57.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T19:59:57.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 9 2026-02-22T19:59:57.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T19:59:57.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T19:59:57.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T19:59:57.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T19:59:57.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 19:58:57 2026 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 19:59:41 2026 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T19:59:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T19:59:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n snap4-clean ']' 2026-02-22T19:59:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap4-clean 2026-02-22T19:59:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap4-clean 2026-02-22T19:59:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T19:59:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:59:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:59:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:59:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:59:58.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:59:58.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:59:58.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:59:58.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:59:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:59:59.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T19:59:59.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap5 2026-02-22T19:59:59.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T19:59:59.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T19:59:59.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap5 2026-02-22T19:59:59.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T19:59:59.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:59:59.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:59:59.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T19:59:59.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T19:59:59.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T19:59:59.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:00:00.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:00:00.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:00:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:00:00.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap5-clean 2026-02-22T20:00:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap5-clean 2026-02-22T20:00:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:00:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap5-clean 2026-02-22T20:00:00.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T20:00:01.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:00:01.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T20:00:01.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:00:01.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:01.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:01.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:00:01.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=9 2026-02-22T20:00:01.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:00:01.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T20:00:03.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 110114816 bytes 2026-02-22T20:00:04.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 137341952 bytes 2026-02-22T20:00:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 149526528 bytes 2026-02-22T20:00:05.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 220183552 bytes 2026-02-22T20:00:05.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 231156736 bytes 2026-02-22T20:00:05.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 257018880 bytes 2026-02-22T20:00:06.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 295276544 bytes 2026-02-22T20:00:07.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 323202048 bytes 2026-02-22T20:00:07.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 357421056 bytes 2026-02-22T20:00:08.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 386358272 bytes 2026-02-22T20:00:08.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 390551552 bytes 2026-02-22T20:00:09.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 437788672 bytes 2026-02-22T20:00:09.638 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:00:09.635+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb9d40 2026-02-22T20:00:10.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:00:10.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:00:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:00:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:00:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:00:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:00:10.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:10.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:00:11.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:00:11.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:00:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:00:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:00:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:00:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:00:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T20:00:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:00:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T20:00:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":8813786.1,"entries_behind_primary":0,"entries_per_second":556.7,"non_primary_position":{"entry_tid":1,"object_number":193,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":193,"tag_tid":15}} 2026-02-22T20:00:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.0 on trial066 2026-02-22T20:00:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:00:03 2026-02-22T20:00:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T20:00:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T20:00:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T20:00:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T20:00:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:00:03' 2026-02-22T20:00:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:00:11.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T20:00:11.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T20:00:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":8813786.1,"entries_behind_primary":0,"entries_per_second":556.7,"non_primary_position":{"entry_tid":1,"object_number":193,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":193,"tag_tid":15}} 2026-02-22T20:00:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:00:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:00:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T20:00:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:00:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:11.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.0 on trial066 2026-02-22T20:00:11.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T20:00:11.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:11.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:00:03' 2026-02-22T20:00:11.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:00:03' 2026-02-22T20:00:11.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:11.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:11.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:11.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 20:00:03' 2026-02-22T20:00:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:00:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:00:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:11.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142024257mirror.0trial06619.2.3-689-gf90f4eaetrueOK42134266mirror.3trial06619.2.3-689-gf90f4eaefalseOK43154318mirror.1trial06619.2.3-689-gf90f4eaefalseOK43894391mirror.2trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":8813786.1,"entries_behind_primary":0,"entries_per_second":556.7,"non_primary_position":{"entry_tid":1,"object_number":193,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":193,"tag_tid":15}}42024257mirror.0trial0662026-02-22 20:00:03cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 20:00:03' 2026-02-22T20:00:11.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T20:00:11.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:11.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:00:03' 2026-02-22T20:00:11.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T20:00:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:00:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T20:00:11.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:00:11.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:11.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:11.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap6 2026-02-22T20:00:11.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap6 2026-02-22T20:00:11.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:00:11.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:11.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:11.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap6 2026-02-22T20:00:11.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:00:28.308 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-22T20:00:28.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-22T20:00:28.308 INFO:tasks.rbd_mirror_thrash:waiting for 9.243543219985096 secs before reviving daemons 2026-02-22T20:00:28.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:00:28.298+0000 7fe80f068640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 22074) UID: 1000 2026-02-22T20:00:37.553 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-22T20:00:37.553 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:00:37.553 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-22T20:00:37.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-22T20:00:37.554 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-22T20:00:37.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-22T20:00:37.556 INFO:tasks.rbd_mirror_thrash:waiting for 117.9169555394104 secs before thrashing 2026-02-22T20:00:42.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:00:42.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:42.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:42.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:42.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:00:42.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:42.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:42.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:42.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:00:42.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:00:42.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:00:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:00:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:00:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:00:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:00:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:00:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:00:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:00:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:00:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:00:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:00:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:00:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:00:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:43.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:43.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:43.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:43.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:43.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:00:43.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:00:43.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:00:43.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:00:43.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:00:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:00:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:00:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:00:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:00:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:00:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:43.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:44.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:44.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:44.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:44.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:44.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=213, tag_tid=17, entry_tid=20373], [object_number=212, tag_tid=17, entry_tid=20372], [object_number=215, tag_tid=17, entry_tid=20371], [object_number=214, tag_tid=17, entry_tid=20370]]], state=connected] 2026-02-22T20:00:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:44.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=213, tag_tid=17, entry_tid=20373], [object_number=212, tag_tid=17, entry_tid=20372], [object_number=215, tag_tid=17, entry_tid=20371], [object_number=214, tag_tid=17, entry_tid=20370]]], state=connected] 2026-02-22T20:00:44.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=213, tag_tid=17, entry_tid=20373' 2026-02-22T20:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=213, tag_tid=17, entry_tid=20373' 2026-02-22T20:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=213, tag_tid=17, entry_tid=20373' '!=' '' 2026-02-22T20:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=213, tag_tid=17, entry_tid=20373' = '' 2026-02-22T20:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=213, tag_tid=17, entry_tid=20373' 2026-02-22T20:00:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=213, tag_tid=17, entry_tid=20373' 2026-02-22T20:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:44.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=213, tag_tid=17, entry_tid=20373' 2026-02-22T20:00:44.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:44.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20373 2026-02-22T20:00:44.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 20373 2026-02-22T20:00:44.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:44.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:44.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:44.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:44.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:44.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:44.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:44.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:44.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:44.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:44.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:44.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:44.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:44.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:44.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:44.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:44.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:44.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:44.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:44.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:44.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:44.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:44.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:44.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:44.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:44.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=212, tag_tid=17, entry_tid=20576], [object_number=215, tag_tid=17, entry_tid=20575], [object_number=214, tag_tid=17, entry_tid=20574], [object_number=213, tag_tid=17, entry_tid=20573]]], state=connected] 2026-02-22T20:00:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:44.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:44.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:44.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:44.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:44.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:44.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:44.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:44.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:44.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:44.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:44.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:44.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=212, tag_tid=17, entry_tid=20576], [object_number=215, tag_tid=17, entry_tid=20575], [object_number=214, tag_tid=17, entry_tid=20574], [object_number=213, tag_tid=17, entry_tid=20573]]], state=connected] 2026-02-22T20:00:44.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=212, tag_tid=17, entry_tid=20576' 2026-02-22T20:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=212, tag_tid=17, entry_tid=20576' 2026-02-22T20:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=212, tag_tid=17, entry_tid=20576' '!=' 'object_number=213, tag_tid=17, entry_tid=20373' 2026-02-22T20:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=212, tag_tid=17, entry_tid=20576' = 'object_number=213, tag_tid=17, entry_tid=20373' 2026-02-22T20:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=212, tag_tid=17, entry_tid=20576' 2026-02-22T20:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=212, tag_tid=17, entry_tid=20576' 2026-02-22T20:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:44.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:44.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=212, tag_tid=17, entry_tid=20576' 2026-02-22T20:00:44.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:44.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20576 2026-02-22T20:00:44.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 20576 2026-02-22T20:00:44.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:44.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:44.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:44.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:44.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:44.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:44.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:44.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:44.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:44.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:44.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:44.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:44.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:44.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:44.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:44.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:44.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:44.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:44.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:44.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:44.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:44.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:44.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:44.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:44.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:44.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:44.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:44.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:44.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:44.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:44.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:44.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:44.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:44.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:44.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=212, tag_tid=17, entry_tid=20780], [object_number=215, tag_tid=17, entry_tid=20779], [object_number=214, tag_tid=17, entry_tid=20778], [object_number=213, tag_tid=17, entry_tid=20777]]], state=connected] 2026-02-22T20:00:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:44.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=212, tag_tid=17, entry_tid=20780], [object_number=215, tag_tid=17, entry_tid=20779], [object_number=214, tag_tid=17, entry_tid=20778], [object_number=213, tag_tid=17, entry_tid=20777]]], state=connected] 2026-02-22T20:00:44.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:44.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=212, tag_tid=17, entry_tid=20780' 2026-02-22T20:00:44.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=212, tag_tid=17, entry_tid=20780' 2026-02-22T20:00:44.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=212, tag_tid=17, entry_tid=20780' '!=' 'object_number=212, tag_tid=17, entry_tid=20576' 2026-02-22T20:00:44.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:44.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=212, tag_tid=17, entry_tid=20780' = 'object_number=212, tag_tid=17, entry_tid=20576' 2026-02-22T20:00:44.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=212, tag_tid=17, entry_tid=20780' 2026-02-22T20:00:44.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:44.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:44.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:44.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=212, tag_tid=17, entry_tid=20780' 2026-02-22T20:00:44.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:44.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=212, tag_tid=17, entry_tid=20780' 2026-02-22T20:00:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20780 2026-02-22T20:00:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 20780 2026-02-22T20:00:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:45.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:45.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:45.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:45.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:45.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:45.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:45.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:45.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:45.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=215, tag_tid=17, entry_tid=20979], [object_number=214, tag_tid=17, entry_tid=20978], [object_number=213, tag_tid=17, entry_tid=20977], [object_number=212, tag_tid=17, entry_tid=20976]]], state=connected] 2026-02-22T20:00:45.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:45.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:45.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:45.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:45.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:45.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:45.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:45.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:45.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:45.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:45.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:45.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=215, tag_tid=17, entry_tid=20979], [object_number=214, tag_tid=17, entry_tid=20978], [object_number=213, tag_tid=17, entry_tid=20977], [object_number=212, tag_tid=17, entry_tid=20976]]], state=connected] 2026-02-22T20:00:45.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=215, tag_tid=17, entry_tid=20979' 2026-02-22T20:00:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=215, tag_tid=17, entry_tid=20979' 2026-02-22T20:00:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=215, tag_tid=17, entry_tid=20979' '!=' 'object_number=212, tag_tid=17, entry_tid=20780' 2026-02-22T20:00:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=215, tag_tid=17, entry_tid=20979' = 'object_number=212, tag_tid=17, entry_tid=20780' 2026-02-22T20:00:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=215, tag_tid=17, entry_tid=20979' 2026-02-22T20:00:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:45.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:45.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=215, tag_tid=17, entry_tid=20979' 2026-02-22T20:00:45.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:45.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:45.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:45.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=215, tag_tid=17, entry_tid=20979' 2026-02-22T20:00:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20979 2026-02-22T20:00:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 20979 2026-02-22T20:00:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:45.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:45.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:45.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:45.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:45.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:45.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:45.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:45.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:45.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:45.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:45.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:45.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:45.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:45.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:45.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:45.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:45.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:45.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:45.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:45.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:45.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:45.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:45.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:45.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=212, tag_tid=17, entry_tid=21176], [object_number=215, tag_tid=17, entry_tid=21175], [object_number=214, tag_tid=17, entry_tid=21174], [object_number=213, tag_tid=17, entry_tid=21173]]], state=connected] 2026-02-22T20:00:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:45.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:45.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=212, tag_tid=17, entry_tid=21176], [object_number=215, tag_tid=17, entry_tid=21175], [object_number=214, tag_tid=17, entry_tid=21174], [object_number=213, tag_tid=17, entry_tid=21173]]], state=connected] 2026-02-22T20:00:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=212, tag_tid=17, entry_tid=21176' 2026-02-22T20:00:45.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=212, tag_tid=17, entry_tid=21176' 2026-02-22T20:00:45.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=212, tag_tid=17, entry_tid=21176' '!=' 'object_number=215, tag_tid=17, entry_tid=20979' 2026-02-22T20:00:45.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:45.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=212, tag_tid=17, entry_tid=21176' = 'object_number=215, tag_tid=17, entry_tid=20979' 2026-02-22T20:00:45.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=212, tag_tid=17, entry_tid=21176' 2026-02-22T20:00:45.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:45.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:45.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:45.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=212, tag_tid=17, entry_tid=21176' 2026-02-22T20:00:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=212, tag_tid=17, entry_tid=21176' 2026-02-22T20:00:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:45.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21176 2026-02-22T20:00:45.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 21176 2026-02-22T20:00:45.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:45.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:45.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:45.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:45.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:45.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:45.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:45.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:45.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:45.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:45.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:45.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:45.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:45.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:45.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:45.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:45.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:45.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=214, tag_tid=17, entry_tid=21366], [object_number=213, tag_tid=17, entry_tid=21365], [object_number=212, tag_tid=17, entry_tid=21364], [object_number=215, tag_tid=17, entry_tid=21363]]], state=connected] 2026-02-22T20:00:45.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:45.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:45.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:45.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:45.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:45.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:45.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:46.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:46.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:46.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=214, tag_tid=17, entry_tid=21366], [object_number=213, tag_tid=17, entry_tid=21365], [object_number=212, tag_tid=17, entry_tid=21364], [object_number=215, tag_tid=17, entry_tid=21363]]], state=connected] 2026-02-22T20:00:46.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=214, tag_tid=17, entry_tid=21366' 2026-02-22T20:00:46.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=214, tag_tid=17, entry_tid=21366' 2026-02-22T20:00:46.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=214, tag_tid=17, entry_tid=21366' '!=' 'object_number=212, tag_tid=17, entry_tid=21176' 2026-02-22T20:00:46.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:46.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=214, tag_tid=17, entry_tid=21366' = 'object_number=212, tag_tid=17, entry_tid=21176' 2026-02-22T20:00:46.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=214, tag_tid=17, entry_tid=21366' 2026-02-22T20:00:46.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:46.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:46.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:46.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=214, tag_tid=17, entry_tid=21366' 2026-02-22T20:00:46.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:46.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:46.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:46.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:46.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:46.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=214, tag_tid=17, entry_tid=21366' 2026-02-22T20:00:46.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:46.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21366 2026-02-22T20:00:46.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 21366 2026-02-22T20:00:46.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:46.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:46.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:46.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:46.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:46.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:46.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:46.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:46.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:46.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:46.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:46.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:46.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:46.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=214, tag_tid=17, entry_tid=21574], [object_number=213, tag_tid=17, entry_tid=21573], [object_number=212, tag_tid=17, entry_tid=21572], [object_number=215, tag_tid=17, entry_tid=21571]]], state=connected] 2026-02-22T20:00:46.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:46.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:46.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:46.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:46.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=214, tag_tid=17, entry_tid=21574], [object_number=213, tag_tid=17, entry_tid=21573], [object_number=212, tag_tid=17, entry_tid=21572], [object_number=215, tag_tid=17, entry_tid=21571]]], state=connected] 2026-02-22T20:00:46.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=214, tag_tid=17, entry_tid=21574' 2026-02-22T20:00:46.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=214, tag_tid=17, entry_tid=21574' 2026-02-22T20:00:46.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=214, tag_tid=17, entry_tid=21574' '!=' 'object_number=214, tag_tid=17, entry_tid=21366' 2026-02-22T20:00:46.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:46.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=214, tag_tid=17, entry_tid=21574' = 'object_number=214, tag_tid=17, entry_tid=21366' 2026-02-22T20:00:46.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=214, tag_tid=17, entry_tid=21574' 2026-02-22T20:00:46.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:46.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:46.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:46.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=214, tag_tid=17, entry_tid=21574' 2026-02-22T20:00:46.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:46.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:46.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:46.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=214, tag_tid=17, entry_tid=21574' 2026-02-22T20:00:46.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:46.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21574 2026-02-22T20:00:46.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 21574 2026-02-22T20:00:46.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:46.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:46.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:46.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:46.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:46.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:46.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:46.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:46.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:46.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:46.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:46.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:46.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:46.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:46.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:46.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:46.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:46.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:46.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:46.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=214, tag_tid=17, entry_tid=21782], [object_number=213, tag_tid=17, entry_tid=21781], [object_number=212, tag_tid=17, entry_tid=21780], [object_number=215, tag_tid=17, entry_tid=21779]]], state=connected] 2026-02-22T20:00:46.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:46.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:46.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:46.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:46.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=214, tag_tid=17, entry_tid=21782], [object_number=213, tag_tid=17, entry_tid=21781], [object_number=212, tag_tid=17, entry_tid=21780], [object_number=215, tag_tid=17, entry_tid=21779]]], state=connected] 2026-02-22T20:00:46.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=214, tag_tid=17, entry_tid=21782' 2026-02-22T20:00:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=214, tag_tid=17, entry_tid=21782' 2026-02-22T20:00:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=214, tag_tid=17, entry_tid=21782' '!=' 'object_number=214, tag_tid=17, entry_tid=21574' 2026-02-22T20:00:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=214, tag_tid=17, entry_tid=21782' = 'object_number=214, tag_tid=17, entry_tid=21574' 2026-02-22T20:00:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=214, tag_tid=17, entry_tid=21782' 2026-02-22T20:00:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:46.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=214, tag_tid=17, entry_tid=21782' 2026-02-22T20:00:46.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:46.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:46.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=214, tag_tid=17, entry_tid=21782' 2026-02-22T20:00:46.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:46.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21782 2026-02-22T20:00:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 21782 2026-02-22T20:00:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:46.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:46.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:46.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:46.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:46.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:46.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:46.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:46.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:46.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:46.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:46.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:46.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:46.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:46.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:46.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:46.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:46.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:46.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:46.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:46.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:46.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:46.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:46.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:46.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:47.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:47.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:47.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:47.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:47.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:47.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:47.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:47.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:47.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=214, tag_tid=17, entry_tid=21990], [object_number=213, tag_tid=17, entry_tid=21989], [object_number=212, tag_tid=17, entry_tid=21988], [object_number=215, tag_tid=17, entry_tid=21987]]], state=connected] 2026-02-22T20:00:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=214, tag_tid=17, entry_tid=21990], [object_number=213, tag_tid=17, entry_tid=21989], [object_number=212, tag_tid=17, entry_tid=21988], [object_number=215, tag_tid=17, entry_tid=21987]]], state=connected] 2026-02-22T20:00:47.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=214, tag_tid=17, entry_tid=21990' 2026-02-22T20:00:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=214, tag_tid=17, entry_tid=21990' 2026-02-22T20:00:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=214, tag_tid=17, entry_tid=21990' '!=' 'object_number=214, tag_tid=17, entry_tid=21782' 2026-02-22T20:00:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=214, tag_tid=17, entry_tid=21990' = 'object_number=214, tag_tid=17, entry_tid=21782' 2026-02-22T20:00:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=214, tag_tid=17, entry_tid=21990' 2026-02-22T20:00:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:47.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:47.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=214, tag_tid=17, entry_tid=21990' 2026-02-22T20:00:47.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:47.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:47.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=214, tag_tid=17, entry_tid=21990' 2026-02-22T20:00:47.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:47.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21990 2026-02-22T20:00:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 21990 2026-02-22T20:00:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:47.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:47.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:47.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:47.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:47.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:47.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:47.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:47.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:47.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:47.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:47.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:47.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:47.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:47.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:47.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:47.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:47.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:47.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:47.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:47.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:47.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:47.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:47.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:47.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:47.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=212, tag_tid=17, entry_tid=22212], [object_number=215, tag_tid=17, entry_tid=22211], [object_number=214, tag_tid=17, entry_tid=22210], [object_number=213, tag_tid=17, entry_tid=22209]]], state=connected] 2026-02-22T20:00:47.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:47.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:47.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:47.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:47.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:47.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:47.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:47.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:47.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:47.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=212, tag_tid=17, entry_tid=22212], [object_number=215, tag_tid=17, entry_tid=22211], [object_number=214, tag_tid=17, entry_tid=22210], [object_number=213, tag_tid=17, entry_tid=22209]]], state=connected] 2026-02-22T20:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=212, tag_tid=17, entry_tid=22212' 2026-02-22T20:00:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=212, tag_tid=17, entry_tid=22212' 2026-02-22T20:00:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=212, tag_tid=17, entry_tid=22212' '!=' 'object_number=214, tag_tid=17, entry_tid=21990' 2026-02-22T20:00:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=212, tag_tid=17, entry_tid=22212' = 'object_number=214, tag_tid=17, entry_tid=21990' 2026-02-22T20:00:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=212, tag_tid=17, entry_tid=22212' 2026-02-22T20:00:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=212, tag_tid=17, entry_tid=22212' 2026-02-22T20:00:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:47.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:47.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:47.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:47.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=212, tag_tid=17, entry_tid=22212' 2026-02-22T20:00:47.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:47.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22212 2026-02-22T20:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 22212 2026-02-22T20:00:47.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:47.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:47.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:47.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:47.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:47.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:47.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:47.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:47.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:47.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:47.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:47.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:47.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=213, tag_tid=17, entry_tid=22429], [object_number=212, tag_tid=17, entry_tid=22428], [object_number=215, tag_tid=17, entry_tid=22427], [object_number=214, tag_tid=17, entry_tid=22426]]], state=connected] 2026-02-22T20:00:47.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 53 2026-02-22T20:00:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=213, tag_tid=17, entry_tid=22429], [object_number=212, tag_tid=17, entry_tid=22428], [object_number=215, tag_tid=17, entry_tid=22427], [object_number=214, tag_tid=17, entry_tid=22426]]], state=connected] 2026-02-22T20:00:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=213, tag_tid=17, entry_tid=22429' 2026-02-22T20:00:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=213, tag_tid=17, entry_tid=22429' 2026-02-22T20:00:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=213, tag_tid=17, entry_tid=22429' '!=' 'object_number=212, tag_tid=17, entry_tid=22212' 2026-02-22T20:00:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=213, tag_tid=17, entry_tid=22429' = 'object_number=212, tag_tid=17, entry_tid=22212' 2026-02-22T20:00:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=213, tag_tid=17, entry_tid=22429' 2026-02-22T20:00:47.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:47.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:47.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:47.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=213, tag_tid=17, entry_tid=22429' 2026-02-22T20:00:47.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:47.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:47.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:47.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:47.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=213, tag_tid=17, entry_tid=22429' 2026-02-22T20:00:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22429 2026-02-22T20:00:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 22429 2026-02-22T20:00:47.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:47.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:47.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:48.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:48.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:48.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:48.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:48.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:48.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:48.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:48.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:48.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:48.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:48.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:48.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:48.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:48.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:48.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:48.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:48.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:48.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:48.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:48.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:48.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:48.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:48.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:48.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:48.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:48.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=22540], [object_number=219, tag_tid=17, entry_tid=22539], [object_number=218, tag_tid=17, entry_tid=22538], [object_number=217, tag_tid=17, entry_tid=22537]]], state=connected] 2026-02-22T20:00:48.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:48.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=22540], [object_number=219, tag_tid=17, entry_tid=22539], [object_number=218, tag_tid=17, entry_tid=22538], [object_number=217, tag_tid=17, entry_tid=22537]]], state=connected] 2026-02-22T20:00:48.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=216, tag_tid=17, entry_tid=22540' 2026-02-22T20:00:48.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=216, tag_tid=17, entry_tid=22540' 2026-02-22T20:00:48.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=22540' '!=' 'object_number=213, tag_tid=17, entry_tid=22429' 2026-02-22T20:00:48.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:48.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=22540' = 'object_number=213, tag_tid=17, entry_tid=22429' 2026-02-22T20:00:48.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=216, tag_tid=17, entry_tid=22540' 2026-02-22T20:00:48.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:48.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:48.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=22540' 2026-02-22T20:00:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:48.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:48.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:48.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:48.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:48.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:48.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:48.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=22540' 2026-02-22T20:00:48.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:48.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:48.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22540 2026-02-22T20:00:48.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 22540 2026-02-22T20:00:48.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:48.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:48.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:48.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:48.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:48.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:48.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:48.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:48.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:48.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:48.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:48.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:48.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:48.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:48.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:48.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:48.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:48.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:48.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:48.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:48.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:48.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:48.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:48.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=22756], [object_number=219, tag_tid=17, entry_tid=22755], [object_number=218, tag_tid=17, entry_tid=22754], [object_number=217, tag_tid=17, entry_tid=22753]]], state=connected] 2026-02-22T20:00:48.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:48.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:48.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:48.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:48.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:48.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:48.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=22756], [object_number=219, tag_tid=17, entry_tid=22755], [object_number=218, tag_tid=17, entry_tid=22754], [object_number=217, tag_tid=17, entry_tid=22753]]], state=connected] 2026-02-22T20:00:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=216, tag_tid=17, entry_tid=22756' 2026-02-22T20:00:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=216, tag_tid=17, entry_tid=22756' 2026-02-22T20:00:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=22756' '!=' 'object_number=216, tag_tid=17, entry_tid=22540' 2026-02-22T20:00:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=22756' = 'object_number=216, tag_tid=17, entry_tid=22540' 2026-02-22T20:00:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=216, tag_tid=17, entry_tid=22756' 2026-02-22T20:00:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:49.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:49.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=22756' 2026-02-22T20:00:49.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:49.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:49.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:49.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:49.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:49.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=22756' 2026-02-22T20:00:49.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:49.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22756 2026-02-22T20:00:49.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 22756 2026-02-22T20:00:49.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:49.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:49.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:49.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:49.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:49.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:49.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:49.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:49.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:49.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:49.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:49.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:49.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:49.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:49.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:49.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:49.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=23205], [object_number=216, tag_tid=17, entry_tid=23204], [object_number=219, tag_tid=17, entry_tid=23203], [object_number=218, tag_tid=17, entry_tid=23202]]], state=connected] 2026-02-22T20:00:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=23205], [object_number=216, tag_tid=17, entry_tid=23204], [object_number=219, tag_tid=17, entry_tid=23203], [object_number=218, tag_tid=17, entry_tid=23202]]], state=connected] 2026-02-22T20:00:49.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=217, tag_tid=17, entry_tid=23205' 2026-02-22T20:00:49.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=217, tag_tid=17, entry_tid=23205' 2026-02-22T20:00:49.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=23205' '!=' 'object_number=216, tag_tid=17, entry_tid=22756' 2026-02-22T20:00:49.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:49.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=23205' = 'object_number=216, tag_tid=17, entry_tid=22756' 2026-02-22T20:00:49.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=217, tag_tid=17, entry_tid=23205' 2026-02-22T20:00:49.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:49.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:49.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:49.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=23205' 2026-02-22T20:00:49.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:49.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:49.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:49.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:49.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:49.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=23205' 2026-02-22T20:00:49.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:49.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23205 2026-02-22T20:00:49.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 23205 2026-02-22T20:00:49.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:49.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:49.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:49.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:49.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:49.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:49.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:49.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:49.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:49.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:49.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:49.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:49.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:49.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:49.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:49.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:49.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:49.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:49.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:49.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:49.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=218, tag_tid=17, entry_tid=23530], [object_number=217, tag_tid=17, entry_tid=23529], [object_number=216, tag_tid=17, entry_tid=23528], [object_number=219, tag_tid=17, entry_tid=23527]]], state=connected] 2026-02-22T20:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=218, tag_tid=17, entry_tid=23530], [object_number=217, tag_tid=17, entry_tid=23529], [object_number=216, tag_tid=17, entry_tid=23528], [object_number=219, tag_tid=17, entry_tid=23527]]], state=connected] 2026-02-22T20:00:49.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=218, tag_tid=17, entry_tid=23530' 2026-02-22T20:00:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=218, tag_tid=17, entry_tid=23530' 2026-02-22T20:00:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=218, tag_tid=17, entry_tid=23530' '!=' 'object_number=217, tag_tid=17, entry_tid=23205' 2026-02-22T20:00:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:49.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=218, tag_tid=17, entry_tid=23530' = 'object_number=217, tag_tid=17, entry_tid=23205' 2026-02-22T20:00:49.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=218, tag_tid=17, entry_tid=23530' 2026-02-22T20:00:49.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:49.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:49.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:49.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=218, tag_tid=17, entry_tid=23530' 2026-02-22T20:00:49.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:49.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:49.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:49.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:49.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:49.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=218, tag_tid=17, entry_tid=23530' 2026-02-22T20:00:49.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:49.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23530 2026-02-22T20:00:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 23530 2026-02-22T20:00:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:49.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:50.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:50.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:50.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:50.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:50.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:50.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:50.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:50.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:50.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:50.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:50.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=23745], [object_number=216, tag_tid=17, entry_tid=23744], [object_number=219, tag_tid=17, entry_tid=23743], [object_number=218, tag_tid=17, entry_tid=23742]]], state=connected] 2026-02-22T20:00:50.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:50.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:50.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:50.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:50.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=23745], [object_number=216, tag_tid=17, entry_tid=23744], [object_number=219, tag_tid=17, entry_tid=23743], [object_number=218, tag_tid=17, entry_tid=23742]]], state=connected] 2026-02-22T20:00:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=217, tag_tid=17, entry_tid=23745' 2026-02-22T20:00:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=217, tag_tid=17, entry_tid=23745' 2026-02-22T20:00:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=23745' '!=' 'object_number=218, tag_tid=17, entry_tid=23530' 2026-02-22T20:00:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=23745' = 'object_number=218, tag_tid=17, entry_tid=23530' 2026-02-22T20:00:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=217, tag_tid=17, entry_tid=23745' 2026-02-22T20:00:50.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:50.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:50.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:50.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:50.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=23745' 2026-02-22T20:00:50.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:50.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:50.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:50.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:50.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:50.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:50.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:50.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=23745' 2026-02-22T20:00:50.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:50.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:50.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23745 2026-02-22T20:00:50.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 23745 2026-02-22T20:00:50.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:50.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:50.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:50.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:50.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:50.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:50.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:50.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:50.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:50.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:50.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:50.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:50.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:50.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:50.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:50.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:50.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:50.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:50.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:50.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:50.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:50.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:50.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:50.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:50.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=23953], [object_number=216, tag_tid=17, entry_tid=23952], [object_number=219, tag_tid=17, entry_tid=23951], [object_number=218, tag_tid=17, entry_tid=23950]]], state=connected] 2026-02-22T20:00:50.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:50.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:50.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:50.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:50.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:50.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=23953], [object_number=216, tag_tid=17, entry_tid=23952], [object_number=219, tag_tid=17, entry_tid=23951], [object_number=218, tag_tid=17, entry_tid=23950]]], state=connected] 2026-02-22T20:00:50.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=217, tag_tid=17, entry_tid=23953' 2026-02-22T20:00:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=217, tag_tid=17, entry_tid=23953' 2026-02-22T20:00:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=23953' '!=' 'object_number=217, tag_tid=17, entry_tid=23745' 2026-02-22T20:00:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=23953' = 'object_number=217, tag_tid=17, entry_tid=23745' 2026-02-22T20:00:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=217, tag_tid=17, entry_tid=23953' 2026-02-22T20:00:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=23953' 2026-02-22T20:00:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:50.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:50.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:50.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:50.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:50.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:50.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=23953' 2026-02-22T20:00:50.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:50.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:50.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23953 2026-02-22T20:00:50.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 23953 2026-02-22T20:00:50.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:50.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:50.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:50.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:50.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:50.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:50.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:50.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:50.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:50.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:50.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:50.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:51.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:51.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:51.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:51.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:51.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:51.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:51.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:51.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:51.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:51.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=219, tag_tid=17, entry_tid=24163], [object_number=218, tag_tid=17, entry_tid=24162], [object_number=217, tag_tid=17, entry_tid=24161], [object_number=216, tag_tid=17, entry_tid=24160]]], state=connected] 2026-02-22T20:00:51.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:51.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:51.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:51.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:51.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:51.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:51.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:51.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:51.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:51.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:51.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:51.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=219, tag_tid=17, entry_tid=24163], [object_number=218, tag_tid=17, entry_tid=24162], [object_number=217, tag_tid=17, entry_tid=24161], [object_number=216, tag_tid=17, entry_tid=24160]]], state=connected] 2026-02-22T20:00:51.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=219, tag_tid=17, entry_tid=24163' 2026-02-22T20:00:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=219, tag_tid=17, entry_tid=24163' 2026-02-22T20:00:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=219, tag_tid=17, entry_tid=24163' '!=' 'object_number=217, tag_tid=17, entry_tid=23953' 2026-02-22T20:00:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=219, tag_tid=17, entry_tid=24163' = 'object_number=217, tag_tid=17, entry_tid=23953' 2026-02-22T20:00:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=219, tag_tid=17, entry_tid=24163' 2026-02-22T20:00:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=219, tag_tid=17, entry_tid=24163' 2026-02-22T20:00:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:51.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:51.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:51.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=219, tag_tid=17, entry_tid=24163' 2026-02-22T20:00:51.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:51.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24163 2026-02-22T20:00:51.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 24163 2026-02-22T20:00:51.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:51.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:51.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:51.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:51.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:51.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:51.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:51.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:51.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:51.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=24352], [object_number=219, tag_tid=17, entry_tid=24351], [object_number=218, tag_tid=17, entry_tid=24350], [object_number=217, tag_tid=17, entry_tid=24349]]], state=connected] 2026-02-22T20:00:51.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:51.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:51.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:51.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:51.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:51.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=24352], [object_number=219, tag_tid=17, entry_tid=24351], [object_number=218, tag_tid=17, entry_tid=24350], [object_number=217, tag_tid=17, entry_tid=24349]]], state=connected] 2026-02-22T20:00:51.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=216, tag_tid=17, entry_tid=24352' 2026-02-22T20:00:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=216, tag_tid=17, entry_tid=24352' 2026-02-22T20:00:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=24352' '!=' 'object_number=219, tag_tid=17, entry_tid=24163' 2026-02-22T20:00:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=24352' = 'object_number=219, tag_tid=17, entry_tid=24163' 2026-02-22T20:00:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=216, tag_tid=17, entry_tid=24352' 2026-02-22T20:00:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:51.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=24352' 2026-02-22T20:00:51.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:51.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:51.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=24352' 2026-02-22T20:00:51.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:51.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24352 2026-02-22T20:00:51.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 24352 2026-02-22T20:00:51.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:51.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:51.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:51.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:51.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:51.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:51.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:51.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:51.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:51.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:51.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:51.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:51.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:51.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:51.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:51.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:51.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:51.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:51.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:51.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:51.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:51.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=24561], [object_number=216, tag_tid=17, entry_tid=24560], [object_number=219, tag_tid=17, entry_tid=24559], [object_number=218, tag_tid=17, entry_tid=24558]]], state=connected] 2026-02-22T20:00:51.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:51.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:51.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:51.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=24561], [object_number=216, tag_tid=17, entry_tid=24560], [object_number=219, tag_tid=17, entry_tid=24559], [object_number=218, tag_tid=17, entry_tid=24558]]], state=connected] 2026-02-22T20:00:51.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:51.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=217, tag_tid=17, entry_tid=24561' 2026-02-22T20:00:51.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=217, tag_tid=17, entry_tid=24561' 2026-02-22T20:00:51.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=24561' '!=' 'object_number=216, tag_tid=17, entry_tid=24352' 2026-02-22T20:00:51.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:51.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=24561' = 'object_number=216, tag_tid=17, entry_tid=24352' 2026-02-22T20:00:51.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=217, tag_tid=17, entry_tid=24561' 2026-02-22T20:00:51.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:51.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:51.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:51.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=24561' 2026-02-22T20:00:51.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:51.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=24561' 2026-02-22T20:00:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:51.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24561 2026-02-22T20:00:51.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 24561 2026-02-22T20:00:51.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:51.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:51.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:52.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:52.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:52.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:52.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:52.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:52.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:52.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:52.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:52.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:52.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:52.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:52.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:52.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:52.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:52.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:52.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:52.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:52.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:52.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:52.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:52.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:52.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=218, tag_tid=17, entry_tid=24766], [object_number=217, tag_tid=17, entry_tid=24765], [object_number=216, tag_tid=17, entry_tid=24764], [object_number=219, tag_tid=17, entry_tid=24763]]], state=connected] 2026-02-22T20:00:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=218, tag_tid=17, entry_tid=24766], [object_number=217, tag_tid=17, entry_tid=24765], [object_number=216, tag_tid=17, entry_tid=24764], [object_number=219, tag_tid=17, entry_tid=24763]]], state=connected] 2026-02-22T20:00:52.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=218, tag_tid=17, entry_tid=24766' 2026-02-22T20:00:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=218, tag_tid=17, entry_tid=24766' 2026-02-22T20:00:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=218, tag_tid=17, entry_tid=24766' '!=' 'object_number=217, tag_tid=17, entry_tid=24561' 2026-02-22T20:00:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=218, tag_tid=17, entry_tid=24766' = 'object_number=217, tag_tid=17, entry_tid=24561' 2026-02-22T20:00:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=218, tag_tid=17, entry_tid=24766' 2026-02-22T20:00:52.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:52.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:52.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:52.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=218, tag_tid=17, entry_tid=24766' 2026-02-22T20:00:52.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:52.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:52.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:52.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:52.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:52.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=218, tag_tid=17, entry_tid=24766' 2026-02-22T20:00:52.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24766 2026-02-22T20:00:52.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 24766 2026-02-22T20:00:52.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:52.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:52.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:52.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:52.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:52.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:52.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:52.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:52.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:52.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:52.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:52.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:52.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:52.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:52.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:52.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:52.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:52.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:52.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:52.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:52.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:52.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:52.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:52.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:52.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:52.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:52.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=24992], [object_number=219, tag_tid=17, entry_tid=24991], [object_number=218, tag_tid=17, entry_tid=24990], [object_number=217, tag_tid=17, entry_tid=24989]]], state=connected] 2026-02-22T20:00:52.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:52.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:52.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:52.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:52.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=24992], [object_number=219, tag_tid=17, entry_tid=24991], [object_number=218, tag_tid=17, entry_tid=24990], [object_number=217, tag_tid=17, entry_tid=24989]]], state=connected] 2026-02-22T20:00:52.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=216, tag_tid=17, entry_tid=24992' 2026-02-22T20:00:52.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=216, tag_tid=17, entry_tid=24992' 2026-02-22T20:00:52.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=24992' '!=' 'object_number=218, tag_tid=17, entry_tid=24766' 2026-02-22T20:00:52.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:52.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=24992' = 'object_number=218, tag_tid=17, entry_tid=24766' 2026-02-22T20:00:52.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=216, tag_tid=17, entry_tid=24992' 2026-02-22T20:00:52.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:52.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:52.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:52.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=24992' 2026-02-22T20:00:52.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:52.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:52.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:52.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:52.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:52.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=24992' 2026-02-22T20:00:52.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:52.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24992 2026-02-22T20:00:52.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 24992 2026-02-22T20:00:52.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:52.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:52.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:52.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:52.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:52.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:52.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:52.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:52.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:52.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:52.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:52.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:52.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:52.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:52.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:52.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=218, tag_tid=17, entry_tid=25206], [object_number=217, tag_tid=17, entry_tid=25205], [object_number=216, tag_tid=17, entry_tid=25204], [object_number=219, tag_tid=17, entry_tid=25203]]], state=connected] 2026-02-22T20:00:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=218, tag_tid=17, entry_tid=25206], [object_number=217, tag_tid=17, entry_tid=25205], [object_number=216, tag_tid=17, entry_tid=25204], [object_number=219, tag_tid=17, entry_tid=25203]]], state=connected] 2026-02-22T20:00:52.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=218, tag_tid=17, entry_tid=25206' 2026-02-22T20:00:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=218, tag_tid=17, entry_tid=25206' 2026-02-22T20:00:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=218, tag_tid=17, entry_tid=25206' '!=' 'object_number=216, tag_tid=17, entry_tid=24992' 2026-02-22T20:00:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=218, tag_tid=17, entry_tid=25206' = 'object_number=216, tag_tid=17, entry_tid=24992' 2026-02-22T20:00:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=218, tag_tid=17, entry_tid=25206' 2026-02-22T20:00:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=218, tag_tid=17, entry_tid=25206' 2026-02-22T20:00:52.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:52.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:52.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:52.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:52.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:52.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=218, tag_tid=17, entry_tid=25206' 2026-02-22T20:00:52.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:52.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:52.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25206 2026-02-22T20:00:52.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 25206 2026-02-22T20:00:52.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:52.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:52.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:53.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:53.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:53.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:53.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:53.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:53.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:53.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:53.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:53.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:53.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:53.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:53.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:53.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:53.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:53.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:53.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:53.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:53.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:53.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:53.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:53.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:53.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:53.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:53.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:53.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:53.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:53.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:53.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:53.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:53.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:53.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:53.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:53.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:53.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=25416], [object_number=219, tag_tid=17, entry_tid=25415], [object_number=218, tag_tid=17, entry_tid=25414], [object_number=217, tag_tid=17, entry_tid=25413]]], state=connected] 2026-02-22T20:00:53.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:53.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:53.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:53.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:53.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:53.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:53.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:53.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:53.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:53.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=25416], [object_number=219, tag_tid=17, entry_tid=25415], [object_number=218, tag_tid=17, entry_tid=25414], [object_number=217, tag_tid=17, entry_tid=25413]]], state=connected] 2026-02-22T20:00:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=216, tag_tid=17, entry_tid=25416' 2026-02-22T20:00:53.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=216, tag_tid=17, entry_tid=25416' 2026-02-22T20:00:53.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=25416' '!=' 'object_number=218, tag_tid=17, entry_tid=25206' 2026-02-22T20:00:53.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:53.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=25416' = 'object_number=218, tag_tid=17, entry_tid=25206' 2026-02-22T20:00:53.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=216, tag_tid=17, entry_tid=25416' 2026-02-22T20:00:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=25416' 2026-02-22T20:00:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:53.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:53.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:53.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:53.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:53.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:53.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=25416' 2026-02-22T20:00:53.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:53.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:53.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25416 2026-02-22T20:00:53.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 25416 2026-02-22T20:00:53.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:53.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:53.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:53.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:53.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:53.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:53.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:53.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:53.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:53.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:53.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=219, tag_tid=17, entry_tid=25551], [object_number=218, tag_tid=17, entry_tid=25550], [object_number=217, tag_tid=17, entry_tid=25549], [object_number=216, tag_tid=17, entry_tid=25548]]], state=connected] 2026-02-22T20:00:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:53.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:53.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:53.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:53.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:53.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=219, tag_tid=17, entry_tid=25551], [object_number=218, tag_tid=17, entry_tid=25550], [object_number=217, tag_tid=17, entry_tid=25549], [object_number=216, tag_tid=17, entry_tid=25548]]], state=connected] 2026-02-22T20:00:53.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:53.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=219, tag_tid=17, entry_tid=25551' 2026-02-22T20:00:53.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=219, tag_tid=17, entry_tid=25551' 2026-02-22T20:00:53.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=219, tag_tid=17, entry_tid=25551' '!=' 'object_number=216, tag_tid=17, entry_tid=25416' 2026-02-22T20:00:53.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:53.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=219, tag_tid=17, entry_tid=25551' = 'object_number=216, tag_tid=17, entry_tid=25416' 2026-02-22T20:00:53.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=219, tag_tid=17, entry_tid=25551' 2026-02-22T20:00:53.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:53.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:53.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:53.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:53.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=219, tag_tid=17, entry_tid=25551' 2026-02-22T20:00:53.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:53.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:53.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:53.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:53.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:53.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=219, tag_tid=17, entry_tid=25551' 2026-02-22T20:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25551 2026-02-22T20:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 25551 2026-02-22T20:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:53.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:53.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:53.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:53.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:53.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:53.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:53.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:53.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:53.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:53.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:53.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:53.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:53.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:53.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:53.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:53.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:53.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:53.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:53.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:53.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:53.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:54.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:54.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:54.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:54.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:54.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:54.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:54.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:54.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:54.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=219, tag_tid=17, entry_tid=25763], [object_number=218, tag_tid=17, entry_tid=25762], [object_number=217, tag_tid=17, entry_tid=25761], [object_number=216, tag_tid=17, entry_tid=25760]]], state=connected] 2026-02-22T20:00:54.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:54.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:54.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:54.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:54.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=219, tag_tid=17, entry_tid=25763], [object_number=218, tag_tid=17, entry_tid=25762], [object_number=217, tag_tid=17, entry_tid=25761], [object_number=216, tag_tid=17, entry_tid=25760]]], state=connected] 2026-02-22T20:00:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=219, tag_tid=17, entry_tid=25763' 2026-02-22T20:00:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=219, tag_tid=17, entry_tid=25763' 2026-02-22T20:00:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=219, tag_tid=17, entry_tid=25763' '!=' 'object_number=219, tag_tid=17, entry_tid=25551' 2026-02-22T20:00:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=219, tag_tid=17, entry_tid=25763' = 'object_number=219, tag_tid=17, entry_tid=25551' 2026-02-22T20:00:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=219, tag_tid=17, entry_tid=25763' 2026-02-22T20:00:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=219, tag_tid=17, entry_tid=25763' 2026-02-22T20:00:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:54.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:54.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:54.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=219, tag_tid=17, entry_tid=25763' 2026-02-22T20:00:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25763 2026-02-22T20:00:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 25763 2026-02-22T20:00:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:54.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:54.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:54.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:54.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=25988], [object_number=219, tag_tid=17, entry_tid=25987], [object_number=218, tag_tid=17, entry_tid=25986], [object_number=217, tag_tid=17, entry_tid=25985]]], state=connected] 2026-02-22T20:00:54.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:54.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:54.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:54.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:54.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:54.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:54.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=25988], [object_number=219, tag_tid=17, entry_tid=25987], [object_number=218, tag_tid=17, entry_tid=25986], [object_number=217, tag_tid=17, entry_tid=25985]]], state=connected] 2026-02-22T20:00:54.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=216, tag_tid=17, entry_tid=25988' 2026-02-22T20:00:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=216, tag_tid=17, entry_tid=25988' 2026-02-22T20:00:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=25988' '!=' 'object_number=219, tag_tid=17, entry_tid=25763' 2026-02-22T20:00:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=25988' = 'object_number=219, tag_tid=17, entry_tid=25763' 2026-02-22T20:00:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=216, tag_tid=17, entry_tid=25988' 2026-02-22T20:00:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:54.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:54.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=25988' 2026-02-22T20:00:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:54.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:54.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:54.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:54.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=25988' 2026-02-22T20:00:54.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:54.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25988 2026-02-22T20:00:54.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 25988 2026-02-22T20:00:54.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:54.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:54.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:54.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:54.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:54.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:54.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:54.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:54.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:54.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:54.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:54.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:54.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:54.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:54.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:54.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=26200], [object_number=219, tag_tid=17, entry_tid=26199], [object_number=218, tag_tid=17, entry_tid=26198], [object_number=217, tag_tid=17, entry_tid=26197]]], state=connected] 2026-02-22T20:00:54.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:54.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:54.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:54.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:54.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:54.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:54.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:54.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=216, tag_tid=17, entry_tid=26200], [object_number=219, tag_tid=17, entry_tid=26199], [object_number=218, tag_tid=17, entry_tid=26198], [object_number=217, tag_tid=17, entry_tid=26197]]], state=connected] 2026-02-22T20:00:54.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:54.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=216, tag_tid=17, entry_tid=26200' 2026-02-22T20:00:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=216, tag_tid=17, entry_tid=26200' 2026-02-22T20:00:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=26200' '!=' 'object_number=216, tag_tid=17, entry_tid=25988' 2026-02-22T20:00:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=216, tag_tid=17, entry_tid=26200' = 'object_number=216, tag_tid=17, entry_tid=25988' 2026-02-22T20:00:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=216, tag_tid=17, entry_tid=26200' 2026-02-22T20:00:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:54.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=26200' 2026-02-22T20:00:54.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:54.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:54.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:54.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:54.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:54.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=216, tag_tid=17, entry_tid=26200' 2026-02-22T20:00:54.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:54.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:54.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26200 2026-02-22T20:00:54.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 26200 2026-02-22T20:00:54.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:54.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:54.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:55.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:55.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:55.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:55.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:55.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:55.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:55.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:55.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:55.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:55.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:55.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:55.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:55.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:55.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:55.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:55.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:55.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:55.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:55.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:55.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:55.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=219, tag_tid=17, entry_tid=26419], [object_number=218, tag_tid=17, entry_tid=26418], [object_number=217, tag_tid=17, entry_tid=26417], [object_number=216, tag_tid=17, entry_tid=26416]]], state=connected] 2026-02-22T20:00:55.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:55.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:55.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:55.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:55.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:55.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:55.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:55.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=219, tag_tid=17, entry_tid=26419], [object_number=218, tag_tid=17, entry_tid=26418], [object_number=217, tag_tid=17, entry_tid=26417], [object_number=216, tag_tid=17, entry_tid=26416]]], state=connected] 2026-02-22T20:00:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=219, tag_tid=17, entry_tid=26419' 2026-02-22T20:00:55.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=219, tag_tid=17, entry_tid=26419' 2026-02-22T20:00:55.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=219, tag_tid=17, entry_tid=26419' '!=' 'object_number=216, tag_tid=17, entry_tid=26200' 2026-02-22T20:00:55.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:55.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=219, tag_tid=17, entry_tid=26419' = 'object_number=216, tag_tid=17, entry_tid=26200' 2026-02-22T20:00:55.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=219, tag_tid=17, entry_tid=26419' 2026-02-22T20:00:55.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:55.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:55.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:55.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=219, tag_tid=17, entry_tid=26419' 2026-02-22T20:00:55.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:55.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:55.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:55.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:55.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:55.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=219, tag_tid=17, entry_tid=26419' 2026-02-22T20:00:55.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:55.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26419 2026-02-22T20:00:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 26419 2026-02-22T20:00:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:55.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:55.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:55.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:55.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:55.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:55.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:55.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:55.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:55.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:55.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:55.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:55.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:55.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:55.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:55.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:55.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:55.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:55.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:55.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:55.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:55.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:55.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:55.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:55.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=26637], [object_number=216, tag_tid=17, entry_tid=26636], [object_number=219, tag_tid=17, entry_tid=26635], [object_number=218, tag_tid=17, entry_tid=26634]]], state=connected] 2026-02-22T20:00:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:55.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:55.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:55.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:55.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:55.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:55.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:55.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 54 2026-02-22T20:00:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=217, tag_tid=17, entry_tid=26637], [object_number=216, tag_tid=17, entry_tid=26636], [object_number=219, tag_tid=17, entry_tid=26635], [object_number=218, tag_tid=17, entry_tid=26634]]], state=connected] 2026-02-22T20:00:55.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=217, tag_tid=17, entry_tid=26637' 2026-02-22T20:00:55.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=217, tag_tid=17, entry_tid=26637' 2026-02-22T20:00:55.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=26637' '!=' 'object_number=219, tag_tid=17, entry_tid=26419' 2026-02-22T20:00:55.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=217, tag_tid=17, entry_tid=26637' = 'object_number=219, tag_tid=17, entry_tid=26419' 2026-02-22T20:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=217, tag_tid=17, entry_tid=26637' 2026-02-22T20:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:55.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:55.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=26637' 2026-02-22T20:00:55.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:55.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:55.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:55.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=217, tag_tid=17, entry_tid=26637' 2026-02-22T20:00:55.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:55.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26637 2026-02-22T20:00:55.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 26637 2026-02-22T20:00:55.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:55.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:55.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:55.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:55.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:55.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:55.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:55.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:55.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:55.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:55.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:55.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=221, tag_tid=17, entry_tid=26761], [object_number=220, tag_tid=17, entry_tid=26760], [object_number=223, tag_tid=17, entry_tid=26759], [object_number=222, tag_tid=17, entry_tid=26758]]], state=connected] 2026-02-22T20:00:55.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:55.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:55.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:55.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:55.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:55.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:55.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:55.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:55.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:55.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=221, tag_tid=17, entry_tid=26761], [object_number=220, tag_tid=17, entry_tid=26760], [object_number=223, tag_tid=17, entry_tid=26759], [object_number=222, tag_tid=17, entry_tid=26758]]], state=connected] 2026-02-22T20:00:55.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=221, tag_tid=17, entry_tid=26761' 2026-02-22T20:00:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=221, tag_tid=17, entry_tid=26761' 2026-02-22T20:00:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=221, tag_tid=17, entry_tid=26761' '!=' 'object_number=217, tag_tid=17, entry_tid=26637' 2026-02-22T20:00:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=221, tag_tid=17, entry_tid=26761' = 'object_number=217, tag_tid=17, entry_tid=26637' 2026-02-22T20:00:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=221, tag_tid=17, entry_tid=26761' 2026-02-22T20:00:55.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:55.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:55.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=17, entry_tid=26761' 2026-02-22T20:00:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:55.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:55.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:55.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:55.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=17, entry_tid=26761' 2026-02-22T20:00:55.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:55.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:55.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26761 2026-02-22T20:00:55.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 26761 2026-02-22T20:00:55.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:55.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:55.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:56.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:56.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:56.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:56.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:56.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:56.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:56.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:56.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:56.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:56.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:56.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:56.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:56.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:56.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:56.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:56.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:56.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:56.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:56.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=223, tag_tid=17, entry_tid=26987], [object_number=222, tag_tid=17, entry_tid=26986], [object_number=221, tag_tid=17, entry_tid=26985], [object_number=220, tag_tid=17, entry_tid=26984]]], state=connected] 2026-02-22T20:00:56.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:56.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:56.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:56.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:56.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:56.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:56.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=223, tag_tid=17, entry_tid=26987], [object_number=222, tag_tid=17, entry_tid=26986], [object_number=221, tag_tid=17, entry_tid=26985], [object_number=220, tag_tid=17, entry_tid=26984]]], state=connected] 2026-02-22T20:00:56.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=223, tag_tid=17, entry_tid=26987' 2026-02-22T20:00:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=223, tag_tid=17, entry_tid=26987' 2026-02-22T20:00:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=223, tag_tid=17, entry_tid=26987' '!=' 'object_number=221, tag_tid=17, entry_tid=26761' 2026-02-22T20:00:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=223, tag_tid=17, entry_tid=26987' = 'object_number=221, tag_tid=17, entry_tid=26761' 2026-02-22T20:00:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=223, tag_tid=17, entry_tid=26987' 2026-02-22T20:00:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:56.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:56.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:56.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=223, tag_tid=17, entry_tid=26987' 2026-02-22T20:00:56.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:56.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:56.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:56.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:56.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:56.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:56.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:56.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=223, tag_tid=17, entry_tid=26987' 2026-02-22T20:00:56.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:56.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:56.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26987 2026-02-22T20:00:56.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 26987 2026-02-22T20:00:56.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:56.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:56.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:56.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:56.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:56.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:56.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:56.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:56.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:56.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:56.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:56.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:56.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:56.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:56.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:56.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:56.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:56.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:56.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=27180], [object_number=223, tag_tid=17, entry_tid=27179], [object_number=222, tag_tid=17, entry_tid=27178], [object_number=221, tag_tid=17, entry_tid=27177]]], state=connected] 2026-02-22T20:00:56.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:56.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:56.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:56.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:56.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:56.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:56.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:56.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:56.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:56.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:56.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=27180], [object_number=223, tag_tid=17, entry_tid=27179], [object_number=222, tag_tid=17, entry_tid=27178], [object_number=221, tag_tid=17, entry_tid=27177]]], state=connected] 2026-02-22T20:00:56.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=220, tag_tid=17, entry_tid=27180' 2026-02-22T20:00:56.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=220, tag_tid=17, entry_tid=27180' 2026-02-22T20:00:56.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=27180' '!=' 'object_number=223, tag_tid=17, entry_tid=26987' 2026-02-22T20:00:56.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:56.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=27180' = 'object_number=223, tag_tid=17, entry_tid=26987' 2026-02-22T20:00:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=220, tag_tid=17, entry_tid=27180' 2026-02-22T20:00:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:56.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:56.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=27180' 2026-02-22T20:00:56.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:56.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:56.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:56.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:56.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:56.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:56.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:56.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=27180' 2026-02-22T20:00:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:56.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27180 2026-02-22T20:00:56.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 27180 2026-02-22T20:00:56.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:56.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:56.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:56.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:56.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:56.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:56.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:56.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:56.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:56.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:56.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:56.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:56.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:56.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:56.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:56.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:56.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:56.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:56.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:56.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:56.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:56.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:56.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:56.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=221, tag_tid=17, entry_tid=27373], [object_number=220, tag_tid=17, entry_tid=27372], [object_number=223, tag_tid=17, entry_tid=27371], [object_number=222, tag_tid=17, entry_tid=27370]]], state=connected] 2026-02-22T20:00:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:57.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:57.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:57.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:57.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:57.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:57.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=221, tag_tid=17, entry_tid=27373], [object_number=220, tag_tid=17, entry_tid=27372], [object_number=223, tag_tid=17, entry_tid=27371], [object_number=222, tag_tid=17, entry_tid=27370]]], state=connected] 2026-02-22T20:00:57.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=221, tag_tid=17, entry_tid=27373' 2026-02-22T20:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=221, tag_tid=17, entry_tid=27373' 2026-02-22T20:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=221, tag_tid=17, entry_tid=27373' '!=' 'object_number=220, tag_tid=17, entry_tid=27180' 2026-02-22T20:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=221, tag_tid=17, entry_tid=27373' = 'object_number=220, tag_tid=17, entry_tid=27180' 2026-02-22T20:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=221, tag_tid=17, entry_tid=27373' 2026-02-22T20:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:57.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=17, entry_tid=27373' 2026-02-22T20:00:57.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:57.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:57.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=17, entry_tid=27373' 2026-02-22T20:00:57.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:57.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27373 2026-02-22T20:00:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 27373 2026-02-22T20:00:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:57.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:57.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:57.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:57.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:57.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:57.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:57.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:57.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:57.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:57.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:57.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:57.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:57.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:57.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:57.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:57.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:57.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:57.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:57.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:57.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:57.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:57.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:57.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:57.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:57.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=27558], [object_number=221, tag_tid=17, entry_tid=27557], [object_number=220, tag_tid=17, entry_tid=27556], [object_number=223, tag_tid=17, entry_tid=27555]]], state=connected] 2026-02-22T20:00:57.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:57.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:57.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:57.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:57.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:57.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:57.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=27558], [object_number=221, tag_tid=17, entry_tid=27557], [object_number=220, tag_tid=17, entry_tid=27556], [object_number=223, tag_tid=17, entry_tid=27555]]], state=connected] 2026-02-22T20:00:57.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=222, tag_tid=17, entry_tid=27558' 2026-02-22T20:00:57.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=222, tag_tid=17, entry_tid=27558' 2026-02-22T20:00:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=27558' '!=' 'object_number=221, tag_tid=17, entry_tid=27373' 2026-02-22T20:00:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=27558' = 'object_number=221, tag_tid=17, entry_tid=27373' 2026-02-22T20:00:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=222, tag_tid=17, entry_tid=27558' 2026-02-22T20:00:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=27558' 2026-02-22T20:00:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:57.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:57.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:57.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=27558' 2026-02-22T20:00:57.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:57.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27558 2026-02-22T20:00:57.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 27558 2026-02-22T20:00:57.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:57.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:57.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:57.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:57.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:57.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:57.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:57.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:57.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:57.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:57.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:57.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:57.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:57.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:57.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:57.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:57.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:57.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:57.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:57.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:57.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:57.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:57.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:57.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:57.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:57.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:57.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=223, tag_tid=17, entry_tid=27775], [object_number=222, tag_tid=17, entry_tid=27774], [object_number=221, tag_tid=17, entry_tid=27773], [object_number=220, tag_tid=17, entry_tid=27772]]], state=connected] 2026-02-22T20:00:57.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:57.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:57.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:57.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:57.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=223, tag_tid=17, entry_tid=27775], [object_number=222, tag_tid=17, entry_tid=27774], [object_number=221, tag_tid=17, entry_tid=27773], [object_number=220, tag_tid=17, entry_tid=27772]]], state=connected] 2026-02-22T20:00:57.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:57.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=223, tag_tid=17, entry_tid=27775' 2026-02-22T20:00:57.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=223, tag_tid=17, entry_tid=27775' 2026-02-22T20:00:57.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=223, tag_tid=17, entry_tid=27775' '!=' 'object_number=222, tag_tid=17, entry_tid=27558' 2026-02-22T20:00:57.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:57.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=223, tag_tid=17, entry_tid=27775' = 'object_number=222, tag_tid=17, entry_tid=27558' 2026-02-22T20:00:57.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=223, tag_tid=17, entry_tid=27775' 2026-02-22T20:00:57.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:57.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:57.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=223, tag_tid=17, entry_tid=27775' 2026-02-22T20:00:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:57.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:57.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:57.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:57.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=223, tag_tid=17, entry_tid=27775' 2026-02-22T20:00:57.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:57.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:57.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27775 2026-02-22T20:00:57.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 27775 2026-02-22T20:00:57.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:57.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:57.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:57.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:58.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:58.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:58.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:58.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:58.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:58.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:58.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:58.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:58.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:58.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:58.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:58.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:58.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:58.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:58.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=221, tag_tid=17, entry_tid=27973], [object_number=220, tag_tid=17, entry_tid=27972], [object_number=223, tag_tid=17, entry_tid=27971], [object_number=222, tag_tid=17, entry_tid=27970]]], state=connected] 2026-02-22T20:00:58.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:58.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:58.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:58.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:58.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:58.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:58.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:58.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:58.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:58.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:58.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=221, tag_tid=17, entry_tid=27973], [object_number=220, tag_tid=17, entry_tid=27972], [object_number=223, tag_tid=17, entry_tid=27971], [object_number=222, tag_tid=17, entry_tid=27970]]], state=connected] 2026-02-22T20:00:58.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=221, tag_tid=17, entry_tid=27973' 2026-02-22T20:00:58.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=221, tag_tid=17, entry_tid=27973' 2026-02-22T20:00:58.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=221, tag_tid=17, entry_tid=27973' '!=' 'object_number=223, tag_tid=17, entry_tid=27775' 2026-02-22T20:00:58.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:58.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=221, tag_tid=17, entry_tid=27973' = 'object_number=223, tag_tid=17, entry_tid=27775' 2026-02-22T20:00:58.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=221, tag_tid=17, entry_tid=27973' 2026-02-22T20:00:58.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:58.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:58.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:58.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=17, entry_tid=27973' 2026-02-22T20:00:58.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:58.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:58.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=17, entry_tid=27973' 2026-02-22T20:00:58.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:58.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27973 2026-02-22T20:00:58.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 27973 2026-02-22T20:00:58.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:58.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:58.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:58.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:58.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:58.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:58.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:58.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:58.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:58.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:58.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:58.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:58.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:58.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:58.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:58.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:58.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:58.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:58.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:58.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:58.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:58.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:58.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:58.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:58.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:58.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:58.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=28170], [object_number=221, tag_tid=17, entry_tid=28169], [object_number=220, tag_tid=17, entry_tid=28168], [object_number=223, tag_tid=17, entry_tid=28167]]], state=connected] 2026-02-22T20:00:58.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:58.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:58.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:58.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:58.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:58.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:58.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:58.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:58.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:58.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=28170], [object_number=221, tag_tid=17, entry_tid=28169], [object_number=220, tag_tid=17, entry_tid=28168], [object_number=223, tag_tid=17, entry_tid=28167]]], state=connected] 2026-02-22T20:00:58.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=222, tag_tid=17, entry_tid=28170' 2026-02-22T20:00:58.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=222, tag_tid=17, entry_tid=28170' 2026-02-22T20:00:58.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=28170' '!=' 'object_number=221, tag_tid=17, entry_tid=27973' 2026-02-22T20:00:58.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:58.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=28170' = 'object_number=221, tag_tid=17, entry_tid=27973' 2026-02-22T20:00:58.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=222, tag_tid=17, entry_tid=28170' 2026-02-22T20:00:58.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:58.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:58.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:58.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=28170' 2026-02-22T20:00:58.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:58.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:58.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:58.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:58.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=28170' 2026-02-22T20:00:58.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:58.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28170 2026-02-22T20:00:58.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 28170 2026-02-22T20:00:58.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:58.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:58.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:58.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:58.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:58.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:58.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:58.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:58.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:58.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:58.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:58.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=28370], [object_number=221, tag_tid=17, entry_tid=28369], [object_number=220, tag_tid=17, entry_tid=28368], [object_number=223, tag_tid=17, entry_tid=28367]]], state=connected] 2026-02-22T20:00:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:58.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:58.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:58.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:58.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:58.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:58.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=28370], [object_number=221, tag_tid=17, entry_tid=28369], [object_number=220, tag_tid=17, entry_tid=28368], [object_number=223, tag_tid=17, entry_tid=28367]]], state=connected] 2026-02-22T20:00:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:58.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=222, tag_tid=17, entry_tid=28370' 2026-02-22T20:00:58.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=222, tag_tid=17, entry_tid=28370' 2026-02-22T20:00:58.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=28370' '!=' 'object_number=222, tag_tid=17, entry_tid=28170' 2026-02-22T20:00:58.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:58.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=28370' = 'object_number=222, tag_tid=17, entry_tid=28170' 2026-02-22T20:00:58.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=222, tag_tid=17, entry_tid=28370' 2026-02-22T20:00:58.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:58.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:58.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:58.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=28370' 2026-02-22T20:00:58.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:58.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:58.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:58.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:58.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:58.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=28370' 2026-02-22T20:00:58.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:58.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:58.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28370 2026-02-22T20:00:58.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 28370 2026-02-22T20:00:58.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:58.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:58.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:59.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:59.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:59.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:59.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:59.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:59.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:59.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:59.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:59.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:59.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:59.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:59.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:59.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:59.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:59.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:59.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=28584], [object_number=223, tag_tid=17, entry_tid=28583], [object_number=222, tag_tid=17, entry_tid=28582], [object_number=221, tag_tid=17, entry_tid=28581]]], state=connected] 2026-02-22T20:00:59.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:59.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:59.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:59.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:59.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:59.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:59.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:59.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=28584], [object_number=223, tag_tid=17, entry_tid=28583], [object_number=222, tag_tid=17, entry_tid=28582], [object_number=221, tag_tid=17, entry_tid=28581]]], state=connected] 2026-02-22T20:00:59.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=220, tag_tid=17, entry_tid=28584' 2026-02-22T20:00:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=220, tag_tid=17, entry_tid=28584' 2026-02-22T20:00:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=28584' '!=' 'object_number=222, tag_tid=17, entry_tid=28370' 2026-02-22T20:00:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=28584' = 'object_number=222, tag_tid=17, entry_tid=28370' 2026-02-22T20:00:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=220, tag_tid=17, entry_tid=28584' 2026-02-22T20:00:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:59.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=28584' 2026-02-22T20:00:59.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:59.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:59.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:59.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:59.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:59.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=28584' 2026-02-22T20:00:59.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:59.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28584 2026-02-22T20:00:59.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 28584 2026-02-22T20:00:59.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:59.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:59.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:59.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:59.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:59.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:59.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:59.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:59.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:59.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:59.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:59.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:59.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:59.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:59.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:59.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:59.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:59.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:59.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=28776], [object_number=223, tag_tid=17, entry_tid=28775], [object_number=222, tag_tid=17, entry_tid=28774], [object_number=221, tag_tid=17, entry_tid=28773]]], state=connected] 2026-02-22T20:00:59.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:59.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:59.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:59.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:59.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:59.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:59.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:59.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=28776], [object_number=223, tag_tid=17, entry_tid=28775], [object_number=222, tag_tid=17, entry_tid=28774], [object_number=221, tag_tid=17, entry_tid=28773]]], state=connected] 2026-02-22T20:00:59.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=220, tag_tid=17, entry_tid=28776' 2026-02-22T20:00:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=220, tag_tid=17, entry_tid=28776' 2026-02-22T20:00:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=28776' '!=' 'object_number=220, tag_tid=17, entry_tid=28584' 2026-02-22T20:00:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=28776' = 'object_number=220, tag_tid=17, entry_tid=28584' 2026-02-22T20:00:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=220, tag_tid=17, entry_tid=28776' 2026-02-22T20:00:59.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:59.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:59.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:59.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=28776' 2026-02-22T20:00:59.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:59.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:59.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:59.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:59.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:59.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=28776' 2026-02-22T20:00:59.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:59.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28776 2026-02-22T20:00:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 28776 2026-02-22T20:00:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:00:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:00:59.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:00:59.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:00:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:59.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:00:59.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:00:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:00:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:00:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:00:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:00:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:00:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:00:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:00:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:00:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:59.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:59.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:59.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:59.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:00:59.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:00:59.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:00:59.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:59.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:59.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:59.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:59.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:59.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:59.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:59.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:59.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=223, tag_tid=17, entry_tid=28971], [object_number=222, tag_tid=17, entry_tid=28970], [object_number=221, tag_tid=17, entry_tid=28969], [object_number=220, tag_tid=17, entry_tid=28968]]], state=connected] 2026-02-22T20:00:59.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:59.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:00:59.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:00:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:00:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:00:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:00:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:00:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:00:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:59.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:00:59.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:00:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:00:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:00:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:00:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=223, tag_tid=17, entry_tid=28971], [object_number=222, tag_tid=17, entry_tid=28970], [object_number=221, tag_tid=17, entry_tid=28969], [object_number=220, tag_tid=17, entry_tid=28968]]], state=connected] 2026-02-22T20:00:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:00:59.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=223, tag_tid=17, entry_tid=28971' 2026-02-22T20:00:59.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=223, tag_tid=17, entry_tid=28971' 2026-02-22T20:00:59.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=223, tag_tid=17, entry_tid=28971' '!=' 'object_number=220, tag_tid=17, entry_tid=28776' 2026-02-22T20:00:59.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:00:59.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=223, tag_tid=17, entry_tid=28971' = 'object_number=220, tag_tid=17, entry_tid=28776' 2026-02-22T20:00:59.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=223, tag_tid=17, entry_tid=28971' 2026-02-22T20:00:59.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:59.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:59.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:00:59.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=223, tag_tid=17, entry_tid=28971' 2026-02-22T20:00:59.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:00:59.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:00:59.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:00:59.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:59.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:00:59.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:00:59.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=223, tag_tid=17, entry_tid=28971' 2026-02-22T20:00:59.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:00:59.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:00.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28971 2026-02-22T20:01:00.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 28971 2026-02-22T20:01:00.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:00.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:00.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:00.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:00.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:00.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:00.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:00.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:00.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:00.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:00.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:00.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:00.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:00.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:00.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:00.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:00.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:00.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:00.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:00.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:00.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:00.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:00.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=29174], [object_number=221, tag_tid=17, entry_tid=29173], [object_number=220, tag_tid=17, entry_tid=29172], [object_number=223, tag_tid=17, entry_tid=29171]]], state=connected] 2026-02-22T20:01:00.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:00.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:00.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:00.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:00.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:00.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:00.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:00.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:00.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:00.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:00.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:00.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:00.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:00.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:00.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=29174], [object_number=221, tag_tid=17, entry_tid=29173], [object_number=220, tag_tid=17, entry_tid=29172], [object_number=223, tag_tid=17, entry_tid=29171]]], state=connected] 2026-02-22T20:01:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=222, tag_tid=17, entry_tid=29174' 2026-02-22T20:01:00.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=222, tag_tid=17, entry_tid=29174' 2026-02-22T20:01:00.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=29174' '!=' 'object_number=223, tag_tid=17, entry_tid=28971' 2026-02-22T20:01:00.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:00.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=29174' = 'object_number=223, tag_tid=17, entry_tid=28971' 2026-02-22T20:01:00.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=222, tag_tid=17, entry_tid=29174' 2026-02-22T20:01:00.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:00.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:00.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:00.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:01:00.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=29174' 2026-02-22T20:01:00.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:00.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:00.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:01:00.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:00.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:00.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:00.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:00.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=29174' 2026-02-22T20:01:00.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:00.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29174 2026-02-22T20:01:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 29174 2026-02-22T20:01:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:00.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:00.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:00.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:00.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:00.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:00.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:00.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:00.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:00.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:00.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:00.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:00.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:00.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:00.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:00.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:00.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:00.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:00.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:00.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:00.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:00.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:00.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:00.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:00.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=29382], [object_number=221, tag_tid=17, entry_tid=29381], [object_number=220, tag_tid=17, entry_tid=29380], [object_number=223, tag_tid=17, entry_tid=29379]]], state=connected] 2026-02-22T20:01:00.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:00.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:00.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:00.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=29382], [object_number=221, tag_tid=17, entry_tid=29381], [object_number=220, tag_tid=17, entry_tid=29380], [object_number=223, tag_tid=17, entry_tid=29379]]], state=connected] 2026-02-22T20:01:00.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:00.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=222, tag_tid=17, entry_tid=29382' 2026-02-22T20:01:00.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=222, tag_tid=17, entry_tid=29382' 2026-02-22T20:01:00.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=29382' '!=' 'object_number=222, tag_tid=17, entry_tid=29174' 2026-02-22T20:01:00.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:00.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=29382' = 'object_number=222, tag_tid=17, entry_tid=29174' 2026-02-22T20:01:00.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=222, tag_tid=17, entry_tid=29382' 2026-02-22T20:01:00.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:00.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:00.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:00.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:01:00.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=29382' 2026-02-22T20:01:00.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:00.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:00.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:01:00.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:00.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:00.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:00.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:00.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=29382' 2026-02-22T20:01:00.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:00.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:00.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29382 2026-02-22T20:01:00.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 29382 2026-02-22T20:01:00.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:00.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:00.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:00.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:00.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:00.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:00.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:00.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:00.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:00.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:00.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:00.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:00.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:00.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:00.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:00.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:00.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:00.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:00.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:00.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:00.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:01.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:01.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:01.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:01.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:01.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:01.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:01.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:01.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:01.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:01.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:01.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=29594], [object_number=221, tag_tid=17, entry_tid=29593], [object_number=220, tag_tid=17, entry_tid=29592], [object_number=223, tag_tid=17, entry_tid=29591]]], state=connected] 2026-02-22T20:01:01.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:01.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:01.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:01.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:01.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:01.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:01.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:01.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:01.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:01.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:01.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:01.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:01.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=29594], [object_number=221, tag_tid=17, entry_tid=29593], [object_number=220, tag_tid=17, entry_tid=29592], [object_number=223, tag_tid=17, entry_tid=29591]]], state=connected] 2026-02-22T20:01:01.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=222, tag_tid=17, entry_tid=29594' 2026-02-22T20:01:01.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=222, tag_tid=17, entry_tid=29594' 2026-02-22T20:01:01.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=29594' '!=' 'object_number=222, tag_tid=17, entry_tid=29382' 2026-02-22T20:01:01.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:01.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=29594' = 'object_number=222, tag_tid=17, entry_tid=29382' 2026-02-22T20:01:01.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=222, tag_tid=17, entry_tid=29594' 2026-02-22T20:01:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:01:01.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=29594' 2026-02-22T20:01:01.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:01.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:01:01.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:01.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:01.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=29594' 2026-02-22T20:01:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29594 2026-02-22T20:01:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 29594 2026-02-22T20:01:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:01.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:01.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:01.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:01.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:01.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:01.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:01.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:01.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:01.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:01.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:01.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:01.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:01.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:01.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=29816], [object_number=223, tag_tid=17, entry_tid=29815], [object_number=222, tag_tid=17, entry_tid=29814], [object_number=221, tag_tid=17, entry_tid=29813]]], state=connected] 2026-02-22T20:01:01.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:01.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:01.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:01.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=29816], [object_number=223, tag_tid=17, entry_tid=29815], [object_number=222, tag_tid=17, entry_tid=29814], [object_number=221, tag_tid=17, entry_tid=29813]]], state=connected] 2026-02-22T20:01:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=220, tag_tid=17, entry_tid=29816' 2026-02-22T20:01:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=220, tag_tid=17, entry_tid=29816' 2026-02-22T20:01:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=29816' '!=' 'object_number=222, tag_tid=17, entry_tid=29594' 2026-02-22T20:01:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=29816' = 'object_number=222, tag_tid=17, entry_tid=29594' 2026-02-22T20:01:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=220, tag_tid=17, entry_tid=29816' 2026-02-22T20:01:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:01.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:01:01.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=29816' 2026-02-22T20:01:01.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:01.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:01:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:01.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=29816' 2026-02-22T20:01:01.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:01.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29816 2026-02-22T20:01:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 29816 2026-02-22T20:01:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:01.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:01.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:01.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:01.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:01.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:01.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:01.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:01.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:01.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:01.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:01.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:01.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:01.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:01.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:01.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:01.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:01.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:01.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=30034], [object_number=221, tag_tid=17, entry_tid=30033], [object_number=220, tag_tid=17, entry_tid=30032], [object_number=223, tag_tid=17, entry_tid=30031]]], state=connected] 2026-02-22T20:01:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:01.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:01.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:01.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:01.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:01.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:01.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:01.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:01.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:01.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:01.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:01.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:01.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:01.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:01.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:01.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=222, tag_tid=17, entry_tid=30034], [object_number=221, tag_tid=17, entry_tid=30033], [object_number=220, tag_tid=17, entry_tid=30032], [object_number=223, tag_tid=17, entry_tid=30031]]], state=connected] 2026-02-22T20:01:01.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=222, tag_tid=17, entry_tid=30034' 2026-02-22T20:01:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=222, tag_tid=17, entry_tid=30034' 2026-02-22T20:01:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=30034' '!=' 'object_number=220, tag_tid=17, entry_tid=29816' 2026-02-22T20:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=222, tag_tid=17, entry_tid=30034' = 'object_number=220, tag_tid=17, entry_tid=29816' 2026-02-22T20:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=222, tag_tid=17, entry_tid=30034' 2026-02-22T20:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:01.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:01.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:01:01.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=30034' 2026-02-22T20:01:01.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:01.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:01:01.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:01.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:01.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:01.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=222, tag_tid=17, entry_tid=30034' 2026-02-22T20:01:01.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:01.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:01.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30034 2026-02-22T20:01:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 30034 2026-02-22T20:01:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:02.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:02.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:02.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:02.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:02.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:02.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:02.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:02.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:02.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:02.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:02.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:02.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:02.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:02.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:02.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:02.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:02.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:02.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=30248], [object_number=223, tag_tid=17, entry_tid=30247], [object_number=222, tag_tid=17, entry_tid=30246], [object_number=221, tag_tid=17, entry_tid=30245]]], state=connected] 2026-02-22T20:01:02.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:02.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:02.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:02.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:02.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:02.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:02.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:02.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:02.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:02.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:02.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:02.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:02.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=30248], [object_number=223, tag_tid=17, entry_tid=30247], [object_number=222, tag_tid=17, entry_tid=30246], [object_number=221, tag_tid=17, entry_tid=30245]]], state=connected] 2026-02-22T20:01:02.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=220, tag_tid=17, entry_tid=30248' 2026-02-22T20:01:02.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=220, tag_tid=17, entry_tid=30248' 2026-02-22T20:01:02.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=30248' '!=' 'object_number=222, tag_tid=17, entry_tid=30034' 2026-02-22T20:01:02.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:02.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=30248' = 'object_number=222, tag_tid=17, entry_tid=30034' 2026-02-22T20:01:02.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=220, tag_tid=17, entry_tid=30248' 2026-02-22T20:01:02.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:02.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:02.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:01:02.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=30248' 2026-02-22T20:01:02.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:02.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:01:02.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:02.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:02.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:02.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=30248' 2026-02-22T20:01:02.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:02.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30248 2026-02-22T20:01:02.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 30248 2026-02-22T20:01:02.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:02.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:02.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:02.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:02.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:02.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:02.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:02.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:02.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:02.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:02.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:02.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:02.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:02.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:02.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:02.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=223, tag_tid=17, entry_tid=30451], [object_number=222, tag_tid=17, entry_tid=30450], [object_number=221, tag_tid=17, entry_tid=30449], [object_number=220, tag_tid=17, entry_tid=30448]]], state=connected] 2026-02-22T20:01:02.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:02.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:02.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:02.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:02.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:02.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=223, tag_tid=17, entry_tid=30451], [object_number=222, tag_tid=17, entry_tid=30450], [object_number=221, tag_tid=17, entry_tid=30449], [object_number=220, tag_tid=17, entry_tid=30448]]], state=connected] 2026-02-22T20:01:02.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=223, tag_tid=17, entry_tid=30451' 2026-02-22T20:01:02.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=223, tag_tid=17, entry_tid=30451' 2026-02-22T20:01:02.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=223, tag_tid=17, entry_tid=30451' '!=' 'object_number=220, tag_tid=17, entry_tid=30248' 2026-02-22T20:01:02.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:02.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=223, tag_tid=17, entry_tid=30451' = 'object_number=220, tag_tid=17, entry_tid=30248' 2026-02-22T20:01:02.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=223, tag_tid=17, entry_tid=30451' 2026-02-22T20:01:02.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:02.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:02.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:01:02.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=223, tag_tid=17, entry_tid=30451' 2026-02-22T20:01:02.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:02.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:01:02.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:02.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:02.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:02.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=223, tag_tid=17, entry_tid=30451' 2026-02-22T20:01:02.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:02.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30451 2026-02-22T20:01:02.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 30451 2026-02-22T20:01:02.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:02.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:02.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:02.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:02.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:02.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:02.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:02.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:02.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:02.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:02.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:02.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:02.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:02.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:02.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:02.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:02.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:02.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:02.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:02.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=30664], [object_number=223, tag_tid=17, entry_tid=30663], [object_number=222, tag_tid=17, entry_tid=30662], [object_number=221, tag_tid=17, entry_tid=30661]]], state=connected] 2026-02-22T20:01:02.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:02.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:02.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:02.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:02.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:02.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:02.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:02.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:02.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:02.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:02.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:02.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:02.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:02.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:02.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:02.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=220, tag_tid=17, entry_tid=30664], [object_number=223, tag_tid=17, entry_tid=30663], [object_number=222, tag_tid=17, entry_tid=30662], [object_number=221, tag_tid=17, entry_tid=30661]]], state=connected] 2026-02-22T20:01:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:02.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=220, tag_tid=17, entry_tid=30664' 2026-02-22T20:01:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=220, tag_tid=17, entry_tid=30664' 2026-02-22T20:01:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=30664' '!=' 'object_number=223, tag_tid=17, entry_tid=30451' 2026-02-22T20:01:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=220, tag_tid=17, entry_tid=30664' = 'object_number=223, tag_tid=17, entry_tid=30451' 2026-02-22T20:01:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=220, tag_tid=17, entry_tid=30664' 2026-02-22T20:01:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=18 2026-02-22T20:01:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=30664' 2026-02-22T20:01:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=17 2026-02-22T20:01:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=220, tag_tid=17, entry_tid=30664' 2026-02-22T20:01:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30664 2026-02-22T20:01:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 18 = 17 -a 1 -le 30664 2026-02-22T20:01:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:03.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:03.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:03.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:03.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:03.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:03.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:03.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:03.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:03.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:03.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:03.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:03.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:03.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:03.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:03.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:03.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:03.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:03.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:03.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:03.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:03.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:03.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:03.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:03.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:03.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:03.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:03.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:03.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:03.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:03.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 55 2026-02-22T20:01:03.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 55 2026-02-22T20:01:03.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:03.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:03.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=221, tag_tid=18, entry_tid=1], [object_number=220, tag_tid=18, entry_tid=0], [object_number=223, tag_tid=17, entry_tid=30683], [object_number=222, tag_tid=17, entry_tid=30682]]], state=connected] 2026-02-22T20:01:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=221, tag_tid=18, entry_tid=1' -a 'object_number=221, tag_tid=18, entry_tid=1' = 'object_number=221, tag_tid=18, entry_tid=1' 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap6 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:01:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:01:04.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2026-02-22T20:01:04.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T20:01:04.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 11 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T20:01:04.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 19:59:58 2026 2026-02-22T20:01:04.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:00:42 2026 2026-02-22T20:01:04.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:01:04.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T20:01:04.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:01:04.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:01:04.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:01:04.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:01:04.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n snap5-clean ']' 2026-02-22T20:01:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap5-clean 2026-02-22T20:01:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap5-clean 2026-02-22T20:01:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:01:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:01:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:04.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:04.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:01:04.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:01:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:05.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:01:05.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:05.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:05.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:05.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap6 2026-02-22T20:01:05.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:05.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:05.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap6 2026-02-22T20:01:05.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:01:05.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:01:05.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:05.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:05.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:01:06.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:01:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:06.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:01:06.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:07.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap6-clean 2026-02-22T20:01:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap6-clean 2026-02-22T20:01:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:01:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap6-clean 2026-02-22T20:01:07.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T20:01:07.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:01:07.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T20:01:07.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:01:07.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:07.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:07.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:01:07.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=6 2026-02-22T20:01:07.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:01:07.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T20:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 20001792 bytes 2026-02-22T20:01:08.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 53067776 bytes 2026-02-22T20:01:08.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 75697152 bytes 2026-02-22T20:01:09.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 135226368 bytes 2026-02-22T20:01:10.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 183045120 bytes 2026-02-22T20:01:11.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 239652864 bytes 2026-02-22T20:01:12.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 284996608 bytes 2026-02-22T20:01:13.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 351532032 bytes 2026-02-22T20:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:01:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:01:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:01:14.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:01:14.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:14.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":8708888.529999999,"entries_behind_primary":20,"entries_per_second":549.3,"non_primary_position":{"entry_tid":30664,"object_number":220,"tag_tid":17},"primary_position":{"entry_tid":1,"object_number":221,"tag_tid":18},"seconds_until_synced":0} 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.0 on trial066 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:01:03 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:01:03' 2026-02-22T20:01:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:01:14.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T20:01:14.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T20:01:14.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":8708888.529999999,"entries_behind_primary":20,"entries_per_second":549.3,"non_primary_position":{"entry_tid":30664,"object_number":220,"tag_tid":17},"primary_position":{"entry_tid":1,"object_number":221,"tag_tid":18},"seconds_until_synced":0} 2026-02-22T20:01:14.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:01:14.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:01:14.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T20:01:14.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:01:14.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:14.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.0 on trial066 2026-02-22T20:01:14.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T20:01:14.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:14.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:01:03' 2026-02-22T20:01:14.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:01:03' 2026-02-22T20:01:14.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:14.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:14.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:14.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:14.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 20:01:03' 2026-02-22T20:01:14.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:01:14.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:01:14.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:14.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142024257mirror.0trial06619.2.3-689-gf90f4eaetrueOK42134266mirror.3trial06619.2.3-689-gf90f4eaefalseOK43154318mirror.1trial06619.2.3-689-gf90f4eaefalseOK43894391mirror.2trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":8708888.529999999,"entries_behind_primary":20,"entries_per_second":549.3,"non_primary_position":{"entry_tid":30664,"object_number":220,"tag_tid":17},"primary_position":{"entry_tid":1,"object_number":221,"tag_tid":18},"seconds_until_synced":0}42024257mirror.0trial0662026-02-22 20:01:03cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 20:01:03' 2026-02-22T20:01:14.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T20:01:14.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:14.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:01:03' 2026-02-22T20:01:14.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:14.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T20:01:14.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:01:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T20:01:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:01:14.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:14.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:14.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap7 2026-02-22T20:01:14.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap7 2026-02-22T20:01:14.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:01:14.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:14.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:14.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap7 2026-02-22T20:01:14.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:01:24.027 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7316018441592694) > weight (0.25) 2026-02-22T20:01:24.028 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8615330119875351) > weight (0.25) 2026-02-22T20:01:24.028 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7334682541466709) > weight (0.25) 2026-02-22T20:01:24.028 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6992010243925714) > weight (0.25) 2026-02-22T20:01:24.028 INFO:tasks.rbd_mirror_thrash:waiting for 75.470048407077 secs before thrashing 2026-02-22T20:01:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:01:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:01:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:44.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:01:44.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:01:44.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:44.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:01:45.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:01:45.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:45.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:45.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:45.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:01:45.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:01:45.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:01:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:01:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:01:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:01:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:01:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:01:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:01:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:01:45.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:45.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:01:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:01:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:01:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:01:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:01:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:45.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:45.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:01:45.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:01:45.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:01:45.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:01:45.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:45.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:46.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:46.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:46.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:46.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:46.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:46.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:46.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:46.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:46.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:46.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:46.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:46.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:46.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:46.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:46.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:46.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:46.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:46.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 60 2026-02-22T20:01:46.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:46.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:46.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:46.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=240, tag_tid=20, entry_tid=20220], [object_number=243, tag_tid=20, entry_tid=20219], [object_number=242, tag_tid=20, entry_tid=20218], [object_number=241, tag_tid=20, entry_tid=20217]]], state=connected] 2026-02-22T20:01:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:46.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 60 2026-02-22T20:01:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=240, tag_tid=20, entry_tid=20220], [object_number=243, tag_tid=20, entry_tid=20219], [object_number=242, tag_tid=20, entry_tid=20218], [object_number=241, tag_tid=20, entry_tid=20217]]], state=connected] 2026-02-22T20:01:46.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=240, tag_tid=20, entry_tid=20220' 2026-02-22T20:01:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=240, tag_tid=20, entry_tid=20220' 2026-02-22T20:01:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=240, tag_tid=20, entry_tid=20220' '!=' '' 2026-02-22T20:01:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=240, tag_tid=20, entry_tid=20220' = '' 2026-02-22T20:01:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=240, tag_tid=20, entry_tid=20220' 2026-02-22T20:01:46.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:46.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:46.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=240, tag_tid=20, entry_tid=20220' 2026-02-22T20:01:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:46.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:46.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:46.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:46.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=240, tag_tid=20, entry_tid=20220' 2026-02-22T20:01:46.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:46.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20220 2026-02-22T20:01:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 20220 2026-02-22T20:01:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:46.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:46.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:46.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:46.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:46.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:46.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:46.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:46.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:46.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:46.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 60 2026-02-22T20:01:46.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:46.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:46.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:46.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=243, tag_tid=20, entry_tid=20439], [object_number=242, tag_tid=20, entry_tid=20438], [object_number=241, tag_tid=20, entry_tid=20437], [object_number=240, tag_tid=20, entry_tid=20436]]], state=connected] 2026-02-22T20:01:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:46.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:46.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:46.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:46.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:46.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 60 2026-02-22T20:01:46.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:46.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:46.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:46.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=243, tag_tid=20, entry_tid=20439], [object_number=242, tag_tid=20, entry_tid=20438], [object_number=241, tag_tid=20, entry_tid=20437], [object_number=240, tag_tid=20, entry_tid=20436]]], state=connected] 2026-02-22T20:01:46.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=243, tag_tid=20, entry_tid=20439' 2026-02-22T20:01:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=243, tag_tid=20, entry_tid=20439' 2026-02-22T20:01:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=243, tag_tid=20, entry_tid=20439' '!=' 'object_number=240, tag_tid=20, entry_tid=20220' 2026-02-22T20:01:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=243, tag_tid=20, entry_tid=20439' = 'object_number=240, tag_tid=20, entry_tid=20220' 2026-02-22T20:01:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=243, tag_tid=20, entry_tid=20439' 2026-02-22T20:01:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=243, tag_tid=20, entry_tid=20439' 2026-02-22T20:01:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:46.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=243, tag_tid=20, entry_tid=20439' 2026-02-22T20:01:46.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20439 2026-02-22T20:01:46.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 20439 2026-02-22T20:01:46.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:46.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:46.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:46.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:46.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:46.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:46.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:46.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:46.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:46.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:46.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:46.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:46.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 60 2026-02-22T20:01:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=240, tag_tid=20, entry_tid=20644], [object_number=243, tag_tid=20, entry_tid=20643], [object_number=242, tag_tid=20, entry_tid=20642], [object_number=241, tag_tid=20, entry_tid=20641]]], state=connected] 2026-02-22T20:01:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:46.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:46.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 60 2026-02-22T20:01:46.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:46.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:46.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:46.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=240, tag_tid=20, entry_tid=20644], [object_number=243, tag_tid=20, entry_tid=20643], [object_number=242, tag_tid=20, entry_tid=20642], [object_number=241, tag_tid=20, entry_tid=20641]]], state=connected] 2026-02-22T20:01:46.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:46.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=240, tag_tid=20, entry_tid=20644' 2026-02-22T20:01:46.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=240, tag_tid=20, entry_tid=20644' 2026-02-22T20:01:46.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=240, tag_tid=20, entry_tid=20644' '!=' 'object_number=243, tag_tid=20, entry_tid=20439' 2026-02-22T20:01:46.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:46.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=240, tag_tid=20, entry_tid=20644' = 'object_number=243, tag_tid=20, entry_tid=20439' 2026-02-22T20:01:46.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=240, tag_tid=20, entry_tid=20644' 2026-02-22T20:01:46.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:46.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:46.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:46.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=240, tag_tid=20, entry_tid=20644' 2026-02-22T20:01:46.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:46.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=240, tag_tid=20, entry_tid=20644' 2026-02-22T20:01:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:46.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20644 2026-02-22T20:01:46.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 20644 2026-02-22T20:01:46.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:46.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:46.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:47.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:47.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 60 2026-02-22T20:01:47.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:47.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:47.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:47.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=240, tag_tid=20, entry_tid=20856], [object_number=243, tag_tid=20, entry_tid=20855], [object_number=242, tag_tid=20, entry_tid=20854], [object_number=241, tag_tid=20, entry_tid=20853]]], state=connected] 2026-02-22T20:01:47.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:47.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 60 2026-02-22T20:01:47.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:47.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:47.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:47.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=240, tag_tid=20, entry_tid=20856], [object_number=243, tag_tid=20, entry_tid=20855], [object_number=242, tag_tid=20, entry_tid=20854], [object_number=241, tag_tid=20, entry_tid=20853]]], state=connected] 2026-02-22T20:01:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=240, tag_tid=20, entry_tid=20856' 2026-02-22T20:01:47.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=240, tag_tid=20, entry_tid=20856' 2026-02-22T20:01:47.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=240, tag_tid=20, entry_tid=20856' '!=' 'object_number=240, tag_tid=20, entry_tid=20644' 2026-02-22T20:01:47.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:47.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=240, tag_tid=20, entry_tid=20856' = 'object_number=240, tag_tid=20, entry_tid=20644' 2026-02-22T20:01:47.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=240, tag_tid=20, entry_tid=20856' 2026-02-22T20:01:47.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:47.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:47.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:47.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=240, tag_tid=20, entry_tid=20856' 2026-02-22T20:01:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=240, tag_tid=20, entry_tid=20856' 2026-02-22T20:01:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20856 2026-02-22T20:01:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 20856 2026-02-22T20:01:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:47.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:47.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:47.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:47.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:47.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:47.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:47.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:47.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:47.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:47.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:47.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:47.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:47.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:47.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:47.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:47.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:47.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=247, tag_tid=20, entry_tid=21027], [object_number=246, tag_tid=20, entry_tid=21026], [object_number=245, tag_tid=20, entry_tid=21025], [object_number=244, tag_tid=20, entry_tid=21024]]], state=connected] 2026-02-22T20:01:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:47.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:47.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:47.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:47.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:47.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:47.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=247, tag_tid=20, entry_tid=21027], [object_number=246, tag_tid=20, entry_tid=21026], [object_number=245, tag_tid=20, entry_tid=21025], [object_number=244, tag_tid=20, entry_tid=21024]]], state=connected] 2026-02-22T20:01:47.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=247, tag_tid=20, entry_tid=21027' 2026-02-22T20:01:47.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=247, tag_tid=20, entry_tid=21027' 2026-02-22T20:01:47.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=247, tag_tid=20, entry_tid=21027' '!=' 'object_number=240, tag_tid=20, entry_tid=20856' 2026-02-22T20:01:47.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:47.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=247, tag_tid=20, entry_tid=21027' = 'object_number=240, tag_tid=20, entry_tid=20856' 2026-02-22T20:01:47.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=247, tag_tid=20, entry_tid=21027' 2026-02-22T20:01:47.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:47.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:47.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:47.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:47.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=247, tag_tid=20, entry_tid=21027' 2026-02-22T20:01:47.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:47.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:47.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:47.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:47.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:47.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:47.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:47.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=247, tag_tid=20, entry_tid=21027' 2026-02-22T20:01:47.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:47.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:47.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21027 2026-02-22T20:01:47.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 21027 2026-02-22T20:01:47.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:47.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:47.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:47.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:47.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:47.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:47.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:47.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:47.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:47.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:47.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:47.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:47.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:47.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:47.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:47.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:47.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=245, tag_tid=20, entry_tid=21241], [object_number=244, tag_tid=20, entry_tid=21240], [object_number=247, tag_tid=20, entry_tid=21239], [object_number=246, tag_tid=20, entry_tid=21238]]], state=connected] 2026-02-22T20:01:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:48.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:48.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:48.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:48.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:48.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:48.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:48.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:48.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:48.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=245, tag_tid=20, entry_tid=21241], [object_number=244, tag_tid=20, entry_tid=21240], [object_number=247, tag_tid=20, entry_tid=21239], [object_number=246, tag_tid=20, entry_tid=21238]]], state=connected] 2026-02-22T20:01:48.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=245, tag_tid=20, entry_tid=21241' 2026-02-22T20:01:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=245, tag_tid=20, entry_tid=21241' 2026-02-22T20:01:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=245, tag_tid=20, entry_tid=21241' '!=' 'object_number=247, tag_tid=20, entry_tid=21027' 2026-02-22T20:01:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=245, tag_tid=20, entry_tid=21241' = 'object_number=247, tag_tid=20, entry_tid=21027' 2026-02-22T20:01:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=245, tag_tid=20, entry_tid=21241' 2026-02-22T20:01:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:48.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:48.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:48.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=20, entry_tid=21241' 2026-02-22T20:01:48.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:48.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=20, entry_tid=21241' 2026-02-22T20:01:48.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:48.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21241 2026-02-22T20:01:48.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 21241 2026-02-22T20:01:48.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:48.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:48.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:48.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:48.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:48.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:48.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:48.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:48.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:48.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:48.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:48.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=247, tag_tid=20, entry_tid=21439], [object_number=246, tag_tid=20, entry_tid=21438], [object_number=245, tag_tid=20, entry_tid=21437], [object_number=244, tag_tid=20, entry_tid=21436]]], state=connected] 2026-02-22T20:01:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:48.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:48.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:48.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:48.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=247, tag_tid=20, entry_tid=21439], [object_number=246, tag_tid=20, entry_tid=21438], [object_number=245, tag_tid=20, entry_tid=21437], [object_number=244, tag_tid=20, entry_tid=21436]]], state=connected] 2026-02-22T20:01:48.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=247, tag_tid=20, entry_tid=21439' 2026-02-22T20:01:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=247, tag_tid=20, entry_tid=21439' 2026-02-22T20:01:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=247, tag_tid=20, entry_tid=21439' '!=' 'object_number=245, tag_tid=20, entry_tid=21241' 2026-02-22T20:01:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=247, tag_tid=20, entry_tid=21439' = 'object_number=245, tag_tid=20, entry_tid=21241' 2026-02-22T20:01:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=247, tag_tid=20, entry_tid=21439' 2026-02-22T20:01:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:48.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=247, tag_tid=20, entry_tid=21439' 2026-02-22T20:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:48.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:48.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:48.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:48.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=247, tag_tid=20, entry_tid=21439' 2026-02-22T20:01:48.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:48.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21439 2026-02-22T20:01:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 21439 2026-02-22T20:01:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:48.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:48.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:48.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:48.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:48.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:48.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:48.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=21630], [object_number=245, tag_tid=20, entry_tid=21629], [object_number=244, tag_tid=20, entry_tid=21628], [object_number=247, tag_tid=20, entry_tid=21627]]], state=connected] 2026-02-22T20:01:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:48.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:48.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:48.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:48.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=21630], [object_number=245, tag_tid=20, entry_tid=21629], [object_number=244, tag_tid=20, entry_tid=21628], [object_number=247, tag_tid=20, entry_tid=21627]]], state=connected] 2026-02-22T20:01:48.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=246, tag_tid=20, entry_tid=21630' 2026-02-22T20:01:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=246, tag_tid=20, entry_tid=21630' 2026-02-22T20:01:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=21630' '!=' 'object_number=247, tag_tid=20, entry_tid=21439' 2026-02-22T20:01:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=21630' = 'object_number=247, tag_tid=20, entry_tid=21439' 2026-02-22T20:01:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=246, tag_tid=20, entry_tid=21630' 2026-02-22T20:01:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:48.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=246, tag_tid=20, entry_tid=21630' 2026-02-22T20:01:48.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:48.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=246, tag_tid=20, entry_tid=21630' 2026-02-22T20:01:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21630 2026-02-22T20:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 21630 2026-02-22T20:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:49.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:49.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:49.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:49.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:49.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:49.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:49.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:49.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:49.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:49.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:49.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:49.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:49.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=21942], [object_number=245, tag_tid=20, entry_tid=21941], [object_number=244, tag_tid=20, entry_tid=21940], [object_number=247, tag_tid=20, entry_tid=21939]]], state=connected] 2026-02-22T20:01:49.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=21942], [object_number=245, tag_tid=20, entry_tid=21941], [object_number=244, tag_tid=20, entry_tid=21940], [object_number=247, tag_tid=20, entry_tid=21939]]], state=connected] 2026-02-22T20:01:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=246, tag_tid=20, entry_tid=21942' 2026-02-22T20:01:49.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=246, tag_tid=20, entry_tid=21942' 2026-02-22T20:01:49.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=21942' '!=' 'object_number=246, tag_tid=20, entry_tid=21630' 2026-02-22T20:01:49.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:49.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=21942' = 'object_number=246, tag_tid=20, entry_tid=21630' 2026-02-22T20:01:49.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=246, tag_tid=20, entry_tid=21942' 2026-02-22T20:01:49.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:49.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:49.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:49.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=246, tag_tid=20, entry_tid=21942' 2026-02-22T20:01:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:49.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:49.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:49.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:49.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:49.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=246, tag_tid=20, entry_tid=21942' 2026-02-22T20:01:49.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21942 2026-02-22T20:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 21942 2026-02-22T20:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:49.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:49.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:49.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:49.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:49.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:49.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:49.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:49.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:49.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:49.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:49.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:49.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:49.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:49.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:49.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:49.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:49.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:49.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:49.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:49.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:49.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:49.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:49.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:49.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:49.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:49.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22122], [object_number=245, tag_tid=20, entry_tid=22121], [object_number=244, tag_tid=20, entry_tid=22120], [object_number=247, tag_tid=20, entry_tid=22119]]], state=connected] 2026-02-22T20:01:49.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:49.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:49.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:49.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:49.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:49.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:49.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:49.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:49.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22122], [object_number=245, tag_tid=20, entry_tid=22121], [object_number=244, tag_tid=20, entry_tid=22120], [object_number=247, tag_tid=20, entry_tid=22119]]], state=connected] 2026-02-22T20:01:49.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:49.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=246, tag_tid=20, entry_tid=22122' 2026-02-22T20:01:49.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=246, tag_tid=20, entry_tid=22122' 2026-02-22T20:01:49.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=22122' '!=' 'object_number=246, tag_tid=20, entry_tid=21942' 2026-02-22T20:01:49.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:49.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=22122' = 'object_number=246, tag_tid=20, entry_tid=21942' 2026-02-22T20:01:49.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=246, tag_tid=20, entry_tid=22122' 2026-02-22T20:01:49.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:49.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:49.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:49.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:49.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=246, tag_tid=20, entry_tid=22122' 2026-02-22T20:01:49.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:49.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:49.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:49.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:49.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:49.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:49.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:49.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=246, tag_tid=20, entry_tid=22122' 2026-02-22T20:01:49.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:49.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:49.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22122 2026-02-22T20:01:49.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 22122 2026-02-22T20:01:49.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:49.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:49.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:49.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:49.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:49.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:49.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:49.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:49.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:49.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:50.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:50.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:50.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:50.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:50.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:50.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:50.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:50.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:50.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:50.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:50.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:50.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:50.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:50.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:50.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:50.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:50.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:50.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:50.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=22298' '!=' 'object_number=246, tag_tid=20, entry_tid=22122' 2026-02-22T20:01:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:01:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=22298' = 'object_number=246, tag_tid=20, entry_tid=22122' 2026-02-22T20:01:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=21 2026-02-22T20:01:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:01:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=20 2026-02-22T20:01:50.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:50.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:50.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:50.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:01:50.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:50.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:01:50.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:01:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22298 2026-02-22T20:01:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 21 = 20 -a 1 -le 22298 2026-02-22T20:01:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:01:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:50.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:50.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:50.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:50.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:50.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:50.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:50.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:50.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:50.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:50.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:50.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:50.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:50.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:50.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:50.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:50.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:50.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:50.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:50.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:50.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:50.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:50.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:50.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:50.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:50.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=22298' '!=' 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:50.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:50.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:01:51.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:51.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:51.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:51.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:51.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:51.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:51.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:51.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:51.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:51.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:51.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:51.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:51.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:51.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:51.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:51.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:51.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:51.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:51.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:51.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:51.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:51.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:51.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:51.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:51.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:51.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:51.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:51.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:51.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:51.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:51.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=22298' '!=' 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:52.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:52.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:52.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:52.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:52.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:52.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:52.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:52.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:52.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:52.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:52.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:52.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:52.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:52.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:52.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:52.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:52.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:52.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:52.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:52.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:52.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:52.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:52.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:52.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:52.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:52.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:52.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:52.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:52.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:52.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:52.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:52.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=22298' '!=' 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:01:53.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:53.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:53.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:53.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:53.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:53.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:53.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:53.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:53.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:53.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:53.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:53.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:53.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:53.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:53.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:53.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:53.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:54.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:54.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:54.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:54.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:54.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:54.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:54.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:54.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:54.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:54.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:54.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:54.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:54.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:54.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:54.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:54.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:54.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:54.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:54.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:54.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:54.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:54.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:54.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:54.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:54.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:54.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:54.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=246, tag_tid=20, entry_tid=22298], [object_number=245, tag_tid=20, entry_tid=22297], [object_number=244, tag_tid=20, entry_tid=22296], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:54.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=246, tag_tid=20, entry_tid=22298' '!=' 'object_number=246, tag_tid=20, entry_tid=22298' 2026-02-22T20:01:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:01:56.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:01:56.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:56.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:56.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:56.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:01:56.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:01:56.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:01:56.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:01:56.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:01:56.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:56.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:56.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:01:56.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:01:56.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:56.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:01:56.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:56.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:01:56.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:56.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:56.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:01:56.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:01:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:01:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:01:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:01:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:01:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:01:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:56.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:01:56.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:56.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:01:56.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:56.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:56.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:56.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:01:56.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:56.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:56.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:56.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:01:56.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:56.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:56.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:56.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:56.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:56.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:56.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:56.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:56.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:56.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:56.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:56.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:56.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:56.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:01:56.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:56.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:56.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:56.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:01:56.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:01:56.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:01:56.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:01:56.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:01:56.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:56.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:01:56.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:56.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:56.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:01:56.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:56.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 61 2026-02-22T20:01:56.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 61 2026-02-22T20:01:56.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:01:56.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:56.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=245, tag_tid=21, entry_tid=1], [object_number=244, tag_tid=21, entry_tid=0], [object_number=246, tag_tid=20, entry_tid=22298], [object_number=247, tag_tid=20, entry_tid=22295]]], state=connected] 2026-02-22T20:01:56.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=245, tag_tid=21, entry_tid=1' -a 'object_number=245, tag_tid=21, entry_tid=1' = 'object_number=245, tag_tid=21, entry_tid=1' 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap7 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:01:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:01:57.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2026-02-22T20:01:57.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T20:01:57.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 13 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:01:04 2026 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:01:43 2026 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:01:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:01:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n snap6-clean ']' 2026-02-22T20:01:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap6-clean 2026-02-22T20:01:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap6-clean 2026-02-22T20:01:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:01:57.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:01:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:01:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:01:57.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:01:58.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:58.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:01:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap7 2026-02-22T20:01:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:01:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:01:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap7 2026-02-22T20:01:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:01:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:01:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:01:59.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:01:59.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:01:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:01:59.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:02:00.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:02:00.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:00.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap7-clean 2026-02-22T20:02:00.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap7-clean 2026-02-22T20:02:00.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:02:00.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:00.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:00.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap7-clean 2026-02-22T20:02:00.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T20:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T20:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:02:00.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T20:02:00.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:02:00.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T20:02:01.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 20675584 bytes 2026-02-22T20:02:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 136342528 bytes 2026-02-22T20:02:03.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 154766336 bytes 2026-02-22T20:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 229252096 bytes 2026-02-22T20:02:04.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 264024064 bytes 2026-02-22T20:02:05.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 316021760 bytes 2026-02-22T20:02:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 329920512 bytes 2026-02-22T20:02:05.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 330533888 bytes 2026-02-22T20:02:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 378866688 bytes 2026-02-22T20:02:07.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 404600832 bytes 2026-02-22T20:02:07.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 434239488 bytes 2026-02-22T20:02:08.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 477452288 bytes 2026-02-22T20:02:08.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 478071808 bytes 2026-02-22T20:02:08.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 490141696 bytes 2026-02-22T20:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 526921728 bytes 2026-02-22T20:02:09.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 578502656 bytes 2026-02-22T20:02:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1700 IOs, 604859392 bytes 2026-02-22T20:02:10.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1800 IOs, 658197504 bytes 2026-02-22T20:02:11.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1900 IOs, 692966400 bytes 2026-02-22T20:02:12.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:2000 IOs, 740810752 bytes 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:12.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:02:13.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:02:13.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:13.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:13.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:13.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:13.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:02:13.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:13.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T20:02:13.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:02:13.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T20:02:13.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":7135341.03,"entries_behind_primary":0,"entries_per_second":443.17,"non_primary_position":{"entry_tid":1,"object_number":245,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":245,"tag_tid":21}} 2026-02-22T20:02:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.0 on trial066 2026-02-22T20:02:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:02:03 2026-02-22T20:02:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T20:02:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T20:02:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T20:02:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T20:02:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:02:03' 2026-02-22T20:02:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:02:13.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T20:02:13.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T20:02:13.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":7135341.03,"entries_behind_primary":0,"entries_per_second":443.17,"non_primary_position":{"entry_tid":1,"object_number":245,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":245,"tag_tid":21}} 2026-02-22T20:02:13.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:02:13.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:02:13.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T20:02:13.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:02:13.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:13.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.0 on trial066 2026-02-22T20:02:13.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T20:02:13.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:02:03' 2026-02-22T20:02:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:02:03' 2026-02-22T20:02:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 20:02:03' 2026-02-22T20:02:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:02:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:02:13.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:13.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142024257mirror.0trial06619.2.3-689-gf90f4eaetrueOK42134266mirror.3trial06619.2.3-689-gf90f4eaefalseOK43154318mirror.1trial06619.2.3-689-gf90f4eaefalseOK43894391mirror.2trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":7135341.03,"entries_behind_primary":0,"entries_per_second":443.17,"non_primary_position":{"entry_tid":1,"object_number":245,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":245,"tag_tid":21}}42024257mirror.0trial0662026-02-22 20:02:03cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 20:02:03' 2026-02-22T20:02:13.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T20:02:13.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:13.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:02:03' 2026-02-22T20:02:13.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:13.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T20:02:13.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:02:13.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T20:02:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:02:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap8 2026-02-22T20:02:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap8 2026-02-22T20:02:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:02:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap8 2026-02-22T20:02:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:02:35.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2830239530630124) > weight (0.25) 2026-02-22T20:02:35.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5649308675040041) > weight (0.25) 2026-02-22T20:02:35.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7930478442270884) > weight (0.25) 2026-02-22T20:02:35.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3204676317485754) > weight (0.25) 2026-02-22T20:02:35.474 INFO:tasks.rbd_mirror_thrash:waiting for 79.60744839695722 secs before thrashing 2026-02-22T20:02:39.498 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.33526032163618613) > weight (0.25) 2026-02-22T20:02:39.499 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.43252941431857617) > weight (0.25) 2026-02-22T20:02:39.499 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2733453899960513) > weight (0.25) 2026-02-22T20:02:39.499 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6432567636900492) > weight (0.25) 2026-02-22T20:02:39.499 INFO:tasks.rbd_mirror_thrash:waiting for 73.74763568249669 secs before thrashing 2026-02-22T20:02:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:02:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:02:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:02:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:02:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:02:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:02:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:02:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:02:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:02:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:02:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:02:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:02:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:02:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:02:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:02:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:02:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:44.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:02:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:02:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:02:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:44.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:02:44.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:44.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:44.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:44.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:44.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:44.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:02:44.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:02:44.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:02:44.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:02:44.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:44.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:44.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:44.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:44.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:44.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:44.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:44.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:44.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:44.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:44.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:44.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=20674], [object_number=265, tag_tid=23, entry_tid=20673], [object_number=264, tag_tid=23, entry_tid=20672], [object_number=267, tag_tid=23, entry_tid=20671]]], state=connected] 2026-02-22T20:02:45.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:45.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:45.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:45.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:45.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:45.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:45.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=20674], [object_number=265, tag_tid=23, entry_tid=20673], [object_number=264, tag_tid=23, entry_tid=20672], [object_number=267, tag_tid=23, entry_tid=20671]]], state=connected] 2026-02-22T20:02:45.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=266, tag_tid=23, entry_tid=20674' 2026-02-22T20:02:45.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=266, tag_tid=23, entry_tid=20674' 2026-02-22T20:02:45.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=20674' '!=' '' 2026-02-22T20:02:45.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:45.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=20674' = '' 2026-02-22T20:02:45.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=266, tag_tid=23, entry_tid=20674' 2026-02-22T20:02:45.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:45.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:45.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:45.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=20674' 2026-02-22T20:02:45.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:45.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:45.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:45.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:45.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:45.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=20674' 2026-02-22T20:02:45.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:45.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20674 2026-02-22T20:02:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 20674 2026-02-22T20:02:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:45.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:45.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:45.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:45.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:45.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:45.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:45.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:45.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:45.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:45.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:45.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:45.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:45.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:45.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:45.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=264, tag_tid=23, entry_tid=20872], [object_number=267, tag_tid=23, entry_tid=20871], [object_number=266, tag_tid=23, entry_tid=20870], [object_number=265, tag_tid=23, entry_tid=20869]]], state=connected] 2026-02-22T20:02:45.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:45.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:45.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:45.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:45.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:45.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:45.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:45.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=264, tag_tid=23, entry_tid=20872], [object_number=267, tag_tid=23, entry_tid=20871], [object_number=266, tag_tid=23, entry_tid=20870], [object_number=265, tag_tid=23, entry_tid=20869]]], state=connected] 2026-02-22T20:02:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=264, tag_tid=23, entry_tid=20872' 2026-02-22T20:02:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=264, tag_tid=23, entry_tid=20872' 2026-02-22T20:02:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=264, tag_tid=23, entry_tid=20872' '!=' 'object_number=266, tag_tid=23, entry_tid=20674' 2026-02-22T20:02:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=264, tag_tid=23, entry_tid=20872' = 'object_number=266, tag_tid=23, entry_tid=20674' 2026-02-22T20:02:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=264, tag_tid=23, entry_tid=20872' 2026-02-22T20:02:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:45.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=264, tag_tid=23, entry_tid=20872' 2026-02-22T20:02:45.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:45.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:45.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=264, tag_tid=23, entry_tid=20872' 2026-02-22T20:02:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=20872 2026-02-22T20:02:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 20872 2026-02-22T20:02:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:45.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:45.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:45.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:45.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:45.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:45.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:45.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:45.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:45.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:45.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:45.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:45.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:45.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:45.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:45.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:45.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:45.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:45.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:45.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=267, tag_tid=23, entry_tid=21067], [object_number=266, tag_tid=23, entry_tid=21066], [object_number=265, tag_tid=23, entry_tid=21065], [object_number=264, tag_tid=23, entry_tid=21064]]], state=connected] 2026-02-22T20:02:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:45.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:45.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:45.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:45.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:45.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=267, tag_tid=23, entry_tid=21067], [object_number=266, tag_tid=23, entry_tid=21066], [object_number=265, tag_tid=23, entry_tid=21065], [object_number=264, tag_tid=23, entry_tid=21064]]], state=connected] 2026-02-22T20:02:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=267, tag_tid=23, entry_tid=21067' 2026-02-22T20:02:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=267, tag_tid=23, entry_tid=21067' 2026-02-22T20:02:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=267, tag_tid=23, entry_tid=21067' '!=' 'object_number=264, tag_tid=23, entry_tid=20872' 2026-02-22T20:02:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=267, tag_tid=23, entry_tid=21067' = 'object_number=264, tag_tid=23, entry_tid=20872' 2026-02-22T20:02:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=267, tag_tid=23, entry_tid=21067' 2026-02-22T20:02:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:45.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=267, tag_tid=23, entry_tid=21067' 2026-02-22T20:02:45.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:45.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:45.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:45.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:45.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:45.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=267, tag_tid=23, entry_tid=21067' 2026-02-22T20:02:45.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:45.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21067 2026-02-22T20:02:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 21067 2026-02-22T20:02:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:46.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:46.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:46.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:46.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:46.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:46.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:46.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:46.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:46.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:46.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:46.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:46.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:46.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:46.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:46.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:46.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:46.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:46.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:46.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:46.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:46.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:46.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:46.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:46.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=265, tag_tid=23, entry_tid=21269], [object_number=264, tag_tid=23, entry_tid=21268], [object_number=267, tag_tid=23, entry_tid=21267], [object_number=266, tag_tid=23, entry_tid=21266]]], state=connected] 2026-02-22T20:02:46.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:46.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:46.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:46.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:46.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:46.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:46.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:46.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:46.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:46.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:46.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=265, tag_tid=23, entry_tid=21269], [object_number=264, tag_tid=23, entry_tid=21268], [object_number=267, tag_tid=23, entry_tid=21267], [object_number=266, tag_tid=23, entry_tid=21266]]], state=connected] 2026-02-22T20:02:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=265, tag_tid=23, entry_tid=21269' 2026-02-22T20:02:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=265, tag_tid=23, entry_tid=21269' 2026-02-22T20:02:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=265, tag_tid=23, entry_tid=21269' '!=' 'object_number=267, tag_tid=23, entry_tid=21067' 2026-02-22T20:02:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=265, tag_tid=23, entry_tid=21269' = 'object_number=267, tag_tid=23, entry_tid=21067' 2026-02-22T20:02:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=265, tag_tid=23, entry_tid=21269' 2026-02-22T20:02:46.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:46.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:46.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=265, tag_tid=23, entry_tid=21269' 2026-02-22T20:02:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:46.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:46.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:46.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:46.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=265, tag_tid=23, entry_tid=21269' 2026-02-22T20:02:46.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:46.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21269 2026-02-22T20:02:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 21269 2026-02-22T20:02:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:46.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:46.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:46.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:46.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:46.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:46.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:46.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:46.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:46.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:46.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:46.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:46.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:46.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:46.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:46.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:46.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=21462], [object_number=265, tag_tid=23, entry_tid=21461], [object_number=264, tag_tid=23, entry_tid=21460], [object_number=267, tag_tid=23, entry_tid=21459]]], state=connected] 2026-02-22T20:02:46.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:46.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:46.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:46.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:46.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:46.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=21462], [object_number=265, tag_tid=23, entry_tid=21461], [object_number=264, tag_tid=23, entry_tid=21460], [object_number=267, tag_tid=23, entry_tid=21459]]], state=connected] 2026-02-22T20:02:46.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=266, tag_tid=23, entry_tid=21462' 2026-02-22T20:02:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=266, tag_tid=23, entry_tid=21462' 2026-02-22T20:02:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=21462' '!=' 'object_number=265, tag_tid=23, entry_tid=21269' 2026-02-22T20:02:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=21462' = 'object_number=265, tag_tid=23, entry_tid=21269' 2026-02-22T20:02:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=266, tag_tid=23, entry_tid=21462' 2026-02-22T20:02:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:46.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=21462' 2026-02-22T20:02:46.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:46.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:46.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:46.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:46.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=21462' 2026-02-22T20:02:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21462 2026-02-22T20:02:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 21462 2026-02-22T20:02:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:46.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:46.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:46.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:46.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:46.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:46.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:46.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:46.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:46.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:46.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:46.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:46.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:46.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:46.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:46.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:46.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:46.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:46.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:46.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:46.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=267, tag_tid=23, entry_tid=21663], [object_number=266, tag_tid=23, entry_tid=21662], [object_number=265, tag_tid=23, entry_tid=21661], [object_number=264, tag_tid=23, entry_tid=21660]]], state=connected] 2026-02-22T20:02:46.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=267, tag_tid=23, entry_tid=21663], [object_number=266, tag_tid=23, entry_tid=21662], [object_number=265, tag_tid=23, entry_tid=21661], [object_number=264, tag_tid=23, entry_tid=21660]]], state=connected] 2026-02-22T20:02:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:46.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=267, tag_tid=23, entry_tid=21663' 2026-02-22T20:02:46.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=267, tag_tid=23, entry_tid=21663' 2026-02-22T20:02:46.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=267, tag_tid=23, entry_tid=21663' '!=' 'object_number=266, tag_tid=23, entry_tid=21462' 2026-02-22T20:02:46.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:46.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=267, tag_tid=23, entry_tid=21663' = 'object_number=266, tag_tid=23, entry_tid=21462' 2026-02-22T20:02:46.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=267, tag_tid=23, entry_tid=21663' 2026-02-22T20:02:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=267, tag_tid=23, entry_tid=21663' 2026-02-22T20:02:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:46.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:46.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:46.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=267, tag_tid=23, entry_tid=21663' 2026-02-22T20:02:46.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:46.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21663 2026-02-22T20:02:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 21663 2026-02-22T20:02:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:47.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:47.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:47.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:47.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:47.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:47.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:47.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:47.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:47.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:47.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:47.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:47.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:47.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:47.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:47.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:47.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:47.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:47.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:47.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:47.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:47.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:47.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:47.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:47.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:47.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:47.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:47.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:47.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:47.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:47.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:47.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:47.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:47.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:47.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:47.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:47.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:47.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:47.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:47.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:47.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:47.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:47.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:47.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:47.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:47.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:47.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:47.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=21850], [object_number=265, tag_tid=23, entry_tid=21849], [object_number=264, tag_tid=23, entry_tid=21848], [object_number=267, tag_tid=23, entry_tid=21847]]], state=connected] 2026-02-22T20:02:47.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:47.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:47.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:47.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:47.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:47.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:47.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=21850], [object_number=265, tag_tid=23, entry_tid=21849], [object_number=264, tag_tid=23, entry_tid=21848], [object_number=267, tag_tid=23, entry_tid=21847]]], state=connected] 2026-02-22T20:02:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=266, tag_tid=23, entry_tid=21850' 2026-02-22T20:02:47.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=266, tag_tid=23, entry_tid=21850' 2026-02-22T20:02:47.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=21850' '!=' 'object_number=267, tag_tid=23, entry_tid=21663' 2026-02-22T20:02:47.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:47.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=21850' = 'object_number=267, tag_tid=23, entry_tid=21663' 2026-02-22T20:02:47.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=266, tag_tid=23, entry_tid=21850' 2026-02-22T20:02:47.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:47.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:47.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:47.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:47.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=21850' 2026-02-22T20:02:47.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:47.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:47.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:47.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:47.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:47.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=21850' 2026-02-22T20:02:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=21850 2026-02-22T20:02:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 21850 2026-02-22T20:02:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:47.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:47.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:47.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:47.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:47.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:47.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:47.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:47.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:47.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:47.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:47.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:47.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:47.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:47.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=267, tag_tid=23, entry_tid=22067], [object_number=266, tag_tid=23, entry_tid=22066], [object_number=265, tag_tid=23, entry_tid=22065], [object_number=264, tag_tid=23, entry_tid=22064]]], state=connected] 2026-02-22T20:02:47.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:47.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=267, tag_tid=23, entry_tid=22067], [object_number=266, tag_tid=23, entry_tid=22066], [object_number=265, tag_tid=23, entry_tid=22065], [object_number=264, tag_tid=23, entry_tid=22064]]], state=connected] 2026-02-22T20:02:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=267, tag_tid=23, entry_tid=22067' 2026-02-22T20:02:47.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=267, tag_tid=23, entry_tid=22067' 2026-02-22T20:02:47.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=267, tag_tid=23, entry_tid=22067' '!=' 'object_number=266, tag_tid=23, entry_tid=21850' 2026-02-22T20:02:47.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=267, tag_tid=23, entry_tid=22067' = 'object_number=266, tag_tid=23, entry_tid=21850' 2026-02-22T20:02:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=267, tag_tid=23, entry_tid=22067' 2026-02-22T20:02:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:47.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:47.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:47.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=267, tag_tid=23, entry_tid=22067' 2026-02-22T20:02:47.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:47.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:47.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:47.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:47.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:47.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:47.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:47.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=267, tag_tid=23, entry_tid=22067' 2026-02-22T20:02:47.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:47.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:47.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22067 2026-02-22T20:02:47.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 22067 2026-02-22T20:02:47.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:47.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:47.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:47.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:47.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:47.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:47.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:47.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:47.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:47.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:47.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:47.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:47.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:47.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:47.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:47.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:47.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:47.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=265, tag_tid=23, entry_tid=22281], [object_number=264, tag_tid=23, entry_tid=22280], [object_number=267, tag_tid=23, entry_tid=22279], [object_number=266, tag_tid=23, entry_tid=22278]]], state=connected] 2026-02-22T20:02:47.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:47.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:47.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:47.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:47.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:47.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:47.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:47.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:47.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:48.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:48.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=265, tag_tid=23, entry_tid=22281], [object_number=264, tag_tid=23, entry_tid=22280], [object_number=267, tag_tid=23, entry_tid=22279], [object_number=266, tag_tid=23, entry_tid=22278]]], state=connected] 2026-02-22T20:02:48.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=265, tag_tid=23, entry_tid=22281' 2026-02-22T20:02:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=265, tag_tid=23, entry_tid=22281' 2026-02-22T20:02:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=265, tag_tid=23, entry_tid=22281' '!=' 'object_number=267, tag_tid=23, entry_tid=22067' 2026-02-22T20:02:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=265, tag_tid=23, entry_tid=22281' = 'object_number=267, tag_tid=23, entry_tid=22067' 2026-02-22T20:02:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=265, tag_tid=23, entry_tid=22281' 2026-02-22T20:02:48.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:48.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:48.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:48.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=265, tag_tid=23, entry_tid=22281' 2026-02-22T20:02:48.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:48.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:48.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:48.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:48.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=265, tag_tid=23, entry_tid=22281' 2026-02-22T20:02:48.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:48.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22281 2026-02-22T20:02:48.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 22281 2026-02-22T20:02:48.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:48.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:48.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:48.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:48.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:48.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:48.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:48.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:48.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:48.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:48.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:48.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:48.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:48.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:48.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:48.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:48.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:48.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:48.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:48.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:48.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:48.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:48.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:48.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=22478], [object_number=265, tag_tid=23, entry_tid=22477], [object_number=264, tag_tid=23, entry_tid=22476], [object_number=267, tag_tid=23, entry_tid=22475]]], state=connected] 2026-02-22T20:02:48.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:48.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:48.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:48.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:48.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:48.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:48.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:48.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:48.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:48.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=22478], [object_number=265, tag_tid=23, entry_tid=22477], [object_number=264, tag_tid=23, entry_tid=22476], [object_number=267, tag_tid=23, entry_tid=22475]]], state=connected] 2026-02-22T20:02:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=266, tag_tid=23, entry_tid=22478' 2026-02-22T20:02:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=266, tag_tid=23, entry_tid=22478' 2026-02-22T20:02:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=22478' '!=' 'object_number=265, tag_tid=23, entry_tid=22281' 2026-02-22T20:02:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=22478' = 'object_number=265, tag_tid=23, entry_tid=22281' 2026-02-22T20:02:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=266, tag_tid=23, entry_tid=22478' 2026-02-22T20:02:48.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:48.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:48.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=22478' 2026-02-22T20:02:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:48.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:48.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=22478' 2026-02-22T20:02:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22478 2026-02-22T20:02:48.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 22478 2026-02-22T20:02:48.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:48.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:48.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:48.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:48.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:48.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:48.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:48.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:48.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:48.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:48.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:48.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:48.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:48.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:48.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:48.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:48.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:48.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:48.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:48.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:48.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:48.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:48.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:48.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:48.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:48.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:48.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:48.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:48.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:48.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:48.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:48.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:48.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:48.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=264, tag_tid=23, entry_tid=22680], [object_number=267, tag_tid=23, entry_tid=22679], [object_number=266, tag_tid=23, entry_tid=22678], [object_number=265, tag_tid=23, entry_tid=22677]]], state=connected] 2026-02-22T20:02:48.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:48.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:48.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:48.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:48.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:48.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:48.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:48.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:48.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:48.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:48.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:48.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:48.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=264, tag_tid=23, entry_tid=22680], [object_number=267, tag_tid=23, entry_tid=22679], [object_number=266, tag_tid=23, entry_tid=22678], [object_number=265, tag_tid=23, entry_tid=22677]]], state=connected] 2026-02-22T20:02:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=264, tag_tid=23, entry_tid=22680' 2026-02-22T20:02:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=264, tag_tid=23, entry_tid=22680' 2026-02-22T20:02:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=264, tag_tid=23, entry_tid=22680' '!=' 'object_number=266, tag_tid=23, entry_tid=22478' 2026-02-22T20:02:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=264, tag_tid=23, entry_tid=22680' = 'object_number=266, tag_tid=23, entry_tid=22478' 2026-02-22T20:02:48.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=264, tag_tid=23, entry_tid=22680' 2026-02-22T20:02:48.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:48.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:48.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:48.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=264, tag_tid=23, entry_tid=22680' 2026-02-22T20:02:48.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:48.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:48.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=264, tag_tid=23, entry_tid=22680' 2026-02-22T20:02:48.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:48.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:48.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22680 2026-02-22T20:02:48.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 22680 2026-02-22T20:02:48.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:48.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:48.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:48.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:48.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:48.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:48.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:48.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:48.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:48.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:48.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:48.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:48.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:48.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:49.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:49.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:49.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:49.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:49.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=22870], [object_number=265, tag_tid=23, entry_tid=22869], [object_number=264, tag_tid=23, entry_tid=22868], [object_number=267, tag_tid=23, entry_tid=22867]]], state=connected] 2026-02-22T20:02:49.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:49.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:49.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:49.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:49.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:49.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:49.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:49.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:49.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:49.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:49.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=22870], [object_number=265, tag_tid=23, entry_tid=22869], [object_number=264, tag_tid=23, entry_tid=22868], [object_number=267, tag_tid=23, entry_tid=22867]]], state=connected] 2026-02-22T20:02:49.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=266, tag_tid=23, entry_tid=22870' 2026-02-22T20:02:49.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=266, tag_tid=23, entry_tid=22870' 2026-02-22T20:02:49.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=22870' '!=' 'object_number=264, tag_tid=23, entry_tid=22680' 2026-02-22T20:02:49.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:49.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=22870' = 'object_number=264, tag_tid=23, entry_tid=22680' 2026-02-22T20:02:49.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=266, tag_tid=23, entry_tid=22870' 2026-02-22T20:02:49.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:49.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:49.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:49.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=22870' 2026-02-22T20:02:49.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:49.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:49.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=22870' 2026-02-22T20:02:49.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:49.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=22870 2026-02-22T20:02:49.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 22870 2026-02-22T20:02:49.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:49.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:49.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:49.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:49.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:49.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:49.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:49.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=23078], [object_number=265, tag_tid=23, entry_tid=23077], [object_number=264, tag_tid=23, entry_tid=23076], [object_number=267, tag_tid=23, entry_tid=23075]]], state=connected] 2026-02-22T20:02:49.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:49.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:49.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:49.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:49.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:49.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:49.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:49.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:49.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=23078], [object_number=265, tag_tid=23, entry_tid=23077], [object_number=264, tag_tid=23, entry_tid=23076], [object_number=267, tag_tid=23, entry_tid=23075]]], state=connected] 2026-02-22T20:02:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=266, tag_tid=23, entry_tid=23078' 2026-02-22T20:02:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=266, tag_tid=23, entry_tid=23078' 2026-02-22T20:02:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=23078' '!=' 'object_number=266, tag_tid=23, entry_tid=22870' 2026-02-22T20:02:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=23078' = 'object_number=266, tag_tid=23, entry_tid=22870' 2026-02-22T20:02:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=266, tag_tid=23, entry_tid=23078' 2026-02-22T20:02:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:49.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=23078' 2026-02-22T20:02:49.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:49.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:49.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:49.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:49.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:49.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=23078' 2026-02-22T20:02:49.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:49.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23078 2026-02-22T20:02:49.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 23078 2026-02-22T20:02:49.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:49.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:49.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:49.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:49.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:49.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:49.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:49.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:49.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:49.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:49.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:49.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:49.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:49.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:49.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:49.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:49.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:49.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:49.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:49.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:49.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=23270], [object_number=265, tag_tid=23, entry_tid=23269], [object_number=264, tag_tid=23, entry_tid=23268], [object_number=267, tag_tid=23, entry_tid=23267]]], state=connected] 2026-02-22T20:02:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:49.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:49.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:49.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:49.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:49.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:49.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:49.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:49.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:49.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=23270], [object_number=265, tag_tid=23, entry_tid=23269], [object_number=264, tag_tid=23, entry_tid=23268], [object_number=267, tag_tid=23, entry_tid=23267]]], state=connected] 2026-02-22T20:02:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:49.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=266, tag_tid=23, entry_tid=23270' 2026-02-22T20:02:49.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=266, tag_tid=23, entry_tid=23270' 2026-02-22T20:02:49.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=23270' '!=' 'object_number=266, tag_tid=23, entry_tid=23078' 2026-02-22T20:02:49.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:49.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=23270' = 'object_number=266, tag_tid=23, entry_tid=23078' 2026-02-22T20:02:49.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=266, tag_tid=23, entry_tid=23270' 2026-02-22T20:02:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=23270' 2026-02-22T20:02:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:49.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=23270' 2026-02-22T20:02:49.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:49.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23270 2026-02-22T20:02:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 23270 2026-02-22T20:02:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:50.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:50.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:50.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:50.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:50.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:50.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:50.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:50.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=23470], [object_number=265, tag_tid=23, entry_tid=23469], [object_number=264, tag_tid=23, entry_tid=23468], [object_number=267, tag_tid=23, entry_tid=23467]]], state=connected] 2026-02-22T20:02:50.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:50.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:50.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:50.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:50.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:50.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 66 2026-02-22T20:02:50.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:50.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:50.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:50.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=266, tag_tid=23, entry_tid=23470], [object_number=265, tag_tid=23, entry_tid=23469], [object_number=264, tag_tid=23, entry_tid=23468], [object_number=267, tag_tid=23, entry_tid=23467]]], state=connected] 2026-02-22T20:02:50.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=266, tag_tid=23, entry_tid=23470' 2026-02-22T20:02:50.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=266, tag_tid=23, entry_tid=23470' 2026-02-22T20:02:50.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=23470' '!=' 'object_number=266, tag_tid=23, entry_tid=23270' 2026-02-22T20:02:50.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:50.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=266, tag_tid=23, entry_tid=23470' = 'object_number=266, tag_tid=23, entry_tid=23270' 2026-02-22T20:02:50.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=266, tag_tid=23, entry_tid=23470' 2026-02-22T20:02:50.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:50.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:50.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:50.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=23470' 2026-02-22T20:02:50.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:50.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:50.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:50.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:50.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:50.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=266, tag_tid=23, entry_tid=23470' 2026-02-22T20:02:50.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:50.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23470 2026-02-22T20:02:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 23470 2026-02-22T20:02:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:50.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:50.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:50.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:50.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:50.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:50.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:50.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:50.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:50.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:50.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:50.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:50.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:50.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:50.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:50.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:50.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:50.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:50.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:50.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:50.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:50.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:50.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:50.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:50.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=23585], [object_number=268, tag_tid=23, entry_tid=23584], [object_number=271, tag_tid=23, entry_tid=23583], [object_number=270, tag_tid=23, entry_tid=23582]]], state=connected] 2026-02-22T20:02:50.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:50.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:50.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:50.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:50.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=23585], [object_number=268, tag_tid=23, entry_tid=23584], [object_number=271, tag_tid=23, entry_tid=23583], [object_number=270, tag_tid=23, entry_tid=23582]]], state=connected] 2026-02-22T20:02:50.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=269, tag_tid=23, entry_tid=23585' 2026-02-22T20:02:50.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=269, tag_tid=23, entry_tid=23585' 2026-02-22T20:02:50.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=23585' '!=' 'object_number=266, tag_tid=23, entry_tid=23470' 2026-02-22T20:02:50.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:50.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=23585' = 'object_number=266, tag_tid=23, entry_tid=23470' 2026-02-22T20:02:50.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=269, tag_tid=23, entry_tid=23585' 2026-02-22T20:02:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=23585' 2026-02-22T20:02:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:50.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:50.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:50.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:50.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=23585' 2026-02-22T20:02:50.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:50.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23585 2026-02-22T20:02:50.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 23585 2026-02-22T20:02:50.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:50.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:50.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:50.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:50.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:50.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:50.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:50.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:50.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:50.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:50.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:50.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:50.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:50.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:50.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:50.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:50.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=23786], [object_number=269, tag_tid=23, entry_tid=23785], [object_number=268, tag_tid=23, entry_tid=23784], [object_number=271, tag_tid=23, entry_tid=23783]]], state=connected] 2026-02-22T20:02:50.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:50.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:50.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:50.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:50.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:50.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:50.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=23786], [object_number=269, tag_tid=23, entry_tid=23785], [object_number=268, tag_tid=23, entry_tid=23784], [object_number=271, tag_tid=23, entry_tid=23783]]], state=connected] 2026-02-22T20:02:50.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:50.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=270, tag_tid=23, entry_tid=23786' 2026-02-22T20:02:50.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=270, tag_tid=23, entry_tid=23786' 2026-02-22T20:02:50.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=23786' '!=' 'object_number=269, tag_tid=23, entry_tid=23585' 2026-02-22T20:02:50.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:50.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=23786' = 'object_number=269, tag_tid=23, entry_tid=23585' 2026-02-22T20:02:50.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=270, tag_tid=23, entry_tid=23786' 2026-02-22T20:02:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:50.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=23786' 2026-02-22T20:02:50.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:50.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:50.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:50.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:50.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:50.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=23786' 2026-02-22T20:02:50.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:50.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:50.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23786 2026-02-22T20:02:50.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 23786 2026-02-22T20:02:50.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:50.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:50.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:51.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:51.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:51.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:51.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:51.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:51.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:51.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:51.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:51.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:51.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:51.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:51.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:51.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=23977], [object_number=268, tag_tid=23, entry_tid=23976], [object_number=271, tag_tid=23, entry_tid=23975], [object_number=270, tag_tid=23, entry_tid=23974]]], state=connected] 2026-02-22T20:02:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:51.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:51.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:51.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:51.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:51.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:51.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:51.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:51.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:51.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:51.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:51.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:51.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:51.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:51.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:51.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:51.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:51.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:51.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=23977], [object_number=268, tag_tid=23, entry_tid=23976], [object_number=271, tag_tid=23, entry_tid=23975], [object_number=270, tag_tid=23, entry_tid=23974]]], state=connected] 2026-02-22T20:02:51.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=269, tag_tid=23, entry_tid=23977' 2026-02-22T20:02:51.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=269, tag_tid=23, entry_tid=23977' 2026-02-22T20:02:51.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=23977' '!=' 'object_number=270, tag_tid=23, entry_tid=23786' 2026-02-22T20:02:51.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:51.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=23977' = 'object_number=270, tag_tid=23, entry_tid=23786' 2026-02-22T20:02:51.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=269, tag_tid=23, entry_tid=23977' 2026-02-22T20:02:51.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:51.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:51.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:51.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:51.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=23977' 2026-02-22T20:02:51.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:51.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:51.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:51.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:51.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:51.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:51.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:51.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=23977' 2026-02-22T20:02:51.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:51.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:51.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23977 2026-02-22T20:02:51.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 23977 2026-02-22T20:02:51.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:51.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:51.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:51.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:51.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:51.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:51.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:51.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:51.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:51.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:51.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:51.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:51.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=24290], [object_number=269, tag_tid=23, entry_tid=24289], [object_number=268, tag_tid=23, entry_tid=24288], [object_number=271, tag_tid=23, entry_tid=24287]]], state=connected] 2026-02-22T20:02:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:51.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:51.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=24290], [object_number=269, tag_tid=23, entry_tid=24289], [object_number=268, tag_tid=23, entry_tid=24288], [object_number=271, tag_tid=23, entry_tid=24287]]], state=connected] 2026-02-22T20:02:51.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=270, tag_tid=23, entry_tid=24290' 2026-02-22T20:02:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=270, tag_tid=23, entry_tid=24290' 2026-02-22T20:02:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=24290' '!=' 'object_number=269, tag_tid=23, entry_tid=23977' 2026-02-22T20:02:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=24290' = 'object_number=269, tag_tid=23, entry_tid=23977' 2026-02-22T20:02:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=270, tag_tid=23, entry_tid=24290' 2026-02-22T20:02:51.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:51.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:51.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:51.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=24290' 2026-02-22T20:02:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:51.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:51.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:51.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:51.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:51.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=24290' 2026-02-22T20:02:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:51.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24290 2026-02-22T20:02:51.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 24290 2026-02-22T20:02:51.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:51.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:51.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:52.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:52.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:52.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:52.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:52.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:52.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:52.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:52.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:52.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=24488], [object_number=271, tag_tid=23, entry_tid=24487], [object_number=270, tag_tid=23, entry_tid=24486], [object_number=269, tag_tid=23, entry_tid=24485]]], state=connected] 2026-02-22T20:02:52.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:52.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:52.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:52.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:52.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:52.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:52.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:52.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:52.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:52.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=24488], [object_number=271, tag_tid=23, entry_tid=24487], [object_number=270, tag_tid=23, entry_tid=24486], [object_number=269, tag_tid=23, entry_tid=24485]]], state=connected] 2026-02-22T20:02:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=268, tag_tid=23, entry_tid=24488' 2026-02-22T20:02:52.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=268, tag_tid=23, entry_tid=24488' 2026-02-22T20:02:52.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=24488' '!=' 'object_number=270, tag_tid=23, entry_tid=24290' 2026-02-22T20:02:52.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:52.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=24488' = 'object_number=270, tag_tid=23, entry_tid=24290' 2026-02-22T20:02:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=268, tag_tid=23, entry_tid=24488' 2026-02-22T20:02:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:52.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=24488' 2026-02-22T20:02:52.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:52.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:52.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:52.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:52.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:52.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=24488' 2026-02-22T20:02:52.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:52.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24488 2026-02-22T20:02:52.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 24488 2026-02-22T20:02:52.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:52.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:52.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:52.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:52.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:52.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:52.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:52.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:52.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:52.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:52.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:52.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:52.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:52.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:52.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:52.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:52.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:52.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:52.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:52.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:52.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:52.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=24694], [object_number=269, tag_tid=23, entry_tid=24693], [object_number=268, tag_tid=23, entry_tid=24692], [object_number=271, tag_tid=23, entry_tid=24691]]], state=connected] 2026-02-22T20:02:52.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:52.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:52.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:52.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:52.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=24694], [object_number=269, tag_tid=23, entry_tid=24693], [object_number=268, tag_tid=23, entry_tid=24692], [object_number=271, tag_tid=23, entry_tid=24691]]], state=connected] 2026-02-22T20:02:52.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=270, tag_tid=23, entry_tid=24694' 2026-02-22T20:02:52.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=270, tag_tid=23, entry_tid=24694' 2026-02-22T20:02:52.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=24694' '!=' 'object_number=268, tag_tid=23, entry_tid=24488' 2026-02-22T20:02:52.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:52.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=24694' = 'object_number=268, tag_tid=23, entry_tid=24488' 2026-02-22T20:02:52.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=270, tag_tid=23, entry_tid=24694' 2026-02-22T20:02:52.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:52.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:52.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:52.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=24694' 2026-02-22T20:02:52.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:52.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:52.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:52.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:52.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:52.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=24694' 2026-02-22T20:02:52.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:52.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24694 2026-02-22T20:02:52.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 24694 2026-02-22T20:02:52.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:52.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:52.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:52.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:52.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:52.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:52.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:52.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:52.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:52.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:52.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:52.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:52.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:52.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:52.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:52.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:52.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:52.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:52.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:52.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:52.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=24890], [object_number=269, tag_tid=23, entry_tid=24889], [object_number=268, tag_tid=23, entry_tid=24888], [object_number=271, tag_tid=23, entry_tid=24887]]], state=connected] 2026-02-22T20:02:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:52.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:52.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:52.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:52.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=24890], [object_number=269, tag_tid=23, entry_tid=24889], [object_number=268, tag_tid=23, entry_tid=24888], [object_number=271, tag_tid=23, entry_tid=24887]]], state=connected] 2026-02-22T20:02:52.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:52.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=270, tag_tid=23, entry_tid=24890' 2026-02-22T20:02:52.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=270, tag_tid=23, entry_tid=24890' 2026-02-22T20:02:52.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=24890' '!=' 'object_number=270, tag_tid=23, entry_tid=24694' 2026-02-22T20:02:52.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:52.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=24890' = 'object_number=270, tag_tid=23, entry_tid=24694' 2026-02-22T20:02:52.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=270, tag_tid=23, entry_tid=24890' 2026-02-22T20:02:52.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:52.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:52.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:52.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=24890' 2026-02-22T20:02:52.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:52.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:52.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:52.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:52.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=24890' 2026-02-22T20:02:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:52.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24890 2026-02-22T20:02:52.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 24890 2026-02-22T20:02:52.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:52.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:52.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:53.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:53.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:53.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:53.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:53.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:53.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:53.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:53.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:53.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:53.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:53.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:53.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:53.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:53.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:53.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:53.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:53.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=25084], [object_number=271, tag_tid=23, entry_tid=25083], [object_number=270, tag_tid=23, entry_tid=25082], [object_number=269, tag_tid=23, entry_tid=25081]]], state=connected] 2026-02-22T20:02:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:53.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:53.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:53.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:53.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:53.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:53.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:53.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:53.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:53.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:53.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:53.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:53.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=25084], [object_number=271, tag_tid=23, entry_tid=25083], [object_number=270, tag_tid=23, entry_tid=25082], [object_number=269, tag_tid=23, entry_tid=25081]]], state=connected] 2026-02-22T20:02:53.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=268, tag_tid=23, entry_tid=25084' 2026-02-22T20:02:53.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=268, tag_tid=23, entry_tid=25084' 2026-02-22T20:02:53.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=25084' '!=' 'object_number=270, tag_tid=23, entry_tid=24890' 2026-02-22T20:02:53.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:53.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=25084' = 'object_number=270, tag_tid=23, entry_tid=24890' 2026-02-22T20:02:53.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=268, tag_tid=23, entry_tid=25084' 2026-02-22T20:02:53.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:53.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:53.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:53.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:53.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=25084' 2026-02-22T20:02:53.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:53.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:53.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:53.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:53.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=25084' 2026-02-22T20:02:53.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:53.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:53.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25084 2026-02-22T20:02:53.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 25084 2026-02-22T20:02:53.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:53.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:53.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:53.449 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:02:53.448+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e096680 2026-02-22T20:02:53.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:53.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:53.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:53.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:53.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:53.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:53.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:53.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:53.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:53.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:53.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=25284], [object_number=271, tag_tid=23, entry_tid=25283], [object_number=270, tag_tid=23, entry_tid=25282], [object_number=269, tag_tid=23, entry_tid=25281]]], state=connected] 2026-02-22T20:02:53.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:53.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:53.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:53.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:53.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:53.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:53.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:53.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:53.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=25284], [object_number=271, tag_tid=23, entry_tid=25283], [object_number=270, tag_tid=23, entry_tid=25282], [object_number=269, tag_tid=23, entry_tid=25281]]], state=connected] 2026-02-22T20:02:53.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=268, tag_tid=23, entry_tid=25284' 2026-02-22T20:02:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=268, tag_tid=23, entry_tid=25284' 2026-02-22T20:02:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=25284' '!=' 'object_number=268, tag_tid=23, entry_tid=25084' 2026-02-22T20:02:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=25284' = 'object_number=268, tag_tid=23, entry_tid=25084' 2026-02-22T20:02:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=268, tag_tid=23, entry_tid=25284' 2026-02-22T20:02:53.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:53.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:53.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:53.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:53.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=25284' 2026-02-22T20:02:53.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:53.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:53.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:53.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:53.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=25284' 2026-02-22T20:02:53.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:53.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:53.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25284 2026-02-22T20:02:53.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 25284 2026-02-22T20:02:53.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:53.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:53.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:53.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:53.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:53.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:53.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:53.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:53.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:53.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:53.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:53.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:54.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:54.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:54.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:54.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=271, tag_tid=23, entry_tid=25483], [object_number=270, tag_tid=23, entry_tid=25482], [object_number=269, tag_tid=23, entry_tid=25481], [object_number=268, tag_tid=23, entry_tid=25480]]], state=connected] 2026-02-22T20:02:54.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:54.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:54.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:54.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:54.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:54.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:54.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:54.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:54.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=271, tag_tid=23, entry_tid=25483], [object_number=270, tag_tid=23, entry_tid=25482], [object_number=269, tag_tid=23, entry_tid=25481], [object_number=268, tag_tid=23, entry_tid=25480]]], state=connected] 2026-02-22T20:02:54.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=271, tag_tid=23, entry_tid=25483' 2026-02-22T20:02:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=271, tag_tid=23, entry_tid=25483' 2026-02-22T20:02:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=271, tag_tid=23, entry_tid=25483' '!=' 'object_number=268, tag_tid=23, entry_tid=25284' 2026-02-22T20:02:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=271, tag_tid=23, entry_tid=25483' = 'object_number=268, tag_tid=23, entry_tid=25284' 2026-02-22T20:02:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=271, tag_tid=23, entry_tid=25483' 2026-02-22T20:02:54.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:54.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:54.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:54.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=271, tag_tid=23, entry_tid=25483' 2026-02-22T20:02:54.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:54.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:54.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:54.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:54.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=271, tag_tid=23, entry_tid=25483' 2026-02-22T20:02:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25483 2026-02-22T20:02:54.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 25483 2026-02-22T20:02:54.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:54.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:54.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:54.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:54.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:54.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:54.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:54.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:54.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:54.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:54.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:54.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:54.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:54.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:54.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:54.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:54.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:54.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:54.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:54.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:54.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:54.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:54.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:54.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:54.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:54.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:54.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:54.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:54.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:54.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:54.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:54.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:54.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=25661], [object_number=268, tag_tid=23, entry_tid=25660], [object_number=271, tag_tid=23, entry_tid=25659], [object_number=270, tag_tid=23, entry_tid=25658]]], state=connected] 2026-02-22T20:02:54.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:54.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:54.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:54.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:54.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:54.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:54.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=25661], [object_number=268, tag_tid=23, entry_tid=25660], [object_number=271, tag_tid=23, entry_tid=25659], [object_number=270, tag_tid=23, entry_tid=25658]]], state=connected] 2026-02-22T20:02:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=269, tag_tid=23, entry_tid=25661' 2026-02-22T20:02:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=269, tag_tid=23, entry_tid=25661' 2026-02-22T20:02:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=25661' '!=' 'object_number=271, tag_tid=23, entry_tid=25483' 2026-02-22T20:02:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=25661' = 'object_number=271, tag_tid=23, entry_tid=25483' 2026-02-22T20:02:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=269, tag_tid=23, entry_tid=25661' 2026-02-22T20:02:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:54.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=25661' 2026-02-22T20:02:54.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:54.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:54.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:54.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:54.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=25661' 2026-02-22T20:02:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25661 2026-02-22T20:02:54.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 25661 2026-02-22T20:02:54.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:54.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:54.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:54.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:54.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:54.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:54.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:54.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:54.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:54.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:54.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:54.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:54.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:54.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:54.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:54.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:54.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:54.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:54.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:54.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:54.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:54.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:54.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=25868], [object_number=271, tag_tid=23, entry_tid=25867], [object_number=270, tag_tid=23, entry_tid=25866], [object_number=269, tag_tid=23, entry_tid=25865]]], state=connected] 2026-02-22T20:02:54.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:54.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:54.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:54.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:54.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=25868], [object_number=271, tag_tid=23, entry_tid=25867], [object_number=270, tag_tid=23, entry_tid=25866], [object_number=269, tag_tid=23, entry_tid=25865]]], state=connected] 2026-02-22T20:02:54.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:54.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=268, tag_tid=23, entry_tid=25868' 2026-02-22T20:02:54.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=268, tag_tid=23, entry_tid=25868' 2026-02-22T20:02:54.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=25868' '!=' 'object_number=269, tag_tid=23, entry_tid=25661' 2026-02-22T20:02:54.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:54.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=25868' = 'object_number=269, tag_tid=23, entry_tid=25661' 2026-02-22T20:02:54.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=268, tag_tid=23, entry_tid=25868' 2026-02-22T20:02:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:54.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:54.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=25868' 2026-02-22T20:02:54.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:54.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:54.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=25868' 2026-02-22T20:02:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:54.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:54.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=25868 2026-02-22T20:02:54.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 25868 2026-02-22T20:02:54.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:54.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:54.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:55.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:55.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:55.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:55.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:55.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:55.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:55.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:55.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:55.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:55.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:55.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:55.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:55.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:55.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:55.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:55.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:55.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:55.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=26088], [object_number=271, tag_tid=23, entry_tid=26087], [object_number=270, tag_tid=23, entry_tid=26086], [object_number=269, tag_tid=23, entry_tid=26085]]], state=connected] 2026-02-22T20:02:55.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:55.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:55.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:55.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:55.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=26088], [object_number=271, tag_tid=23, entry_tid=26087], [object_number=270, tag_tid=23, entry_tid=26086], [object_number=269, tag_tid=23, entry_tid=26085]]], state=connected] 2026-02-22T20:02:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=268, tag_tid=23, entry_tid=26088' 2026-02-22T20:02:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=268, tag_tid=23, entry_tid=26088' 2026-02-22T20:02:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=26088' '!=' 'object_number=268, tag_tid=23, entry_tid=25868' 2026-02-22T20:02:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=26088' = 'object_number=268, tag_tid=23, entry_tid=25868' 2026-02-22T20:02:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=268, tag_tid=23, entry_tid=26088' 2026-02-22T20:02:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:55.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:55.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=26088' 2026-02-22T20:02:55.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:55.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:55.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:55.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:55.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:55.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:55.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=26088' 2026-02-22T20:02:55.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:55.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:55.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26088 2026-02-22T20:02:55.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 26088 2026-02-22T20:02:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:55.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:55.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:55.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:55.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:55.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:55.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:55.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:55.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:55.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:55.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:55.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:55.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:55.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:55.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:55.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:55.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:55.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:55.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:55.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:55.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:55.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:55.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=26297], [object_number=268, tag_tid=23, entry_tid=26296], [object_number=271, tag_tid=23, entry_tid=26295], [object_number=270, tag_tid=23, entry_tid=26294]]], state=connected] 2026-02-22T20:02:55.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:55.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:55.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:55.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:55.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:55.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:55.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:55.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:55.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:55.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:55.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:55.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:55.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=26297], [object_number=268, tag_tid=23, entry_tid=26296], [object_number=271, tag_tid=23, entry_tid=26295], [object_number=270, tag_tid=23, entry_tid=26294]]], state=connected] 2026-02-22T20:02:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=269, tag_tid=23, entry_tid=26297' 2026-02-22T20:02:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=269, tag_tid=23, entry_tid=26297' 2026-02-22T20:02:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=26297' '!=' 'object_number=268, tag_tid=23, entry_tid=26088' 2026-02-22T20:02:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=26297' = 'object_number=268, tag_tid=23, entry_tid=26088' 2026-02-22T20:02:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=269, tag_tid=23, entry_tid=26297' 2026-02-22T20:02:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:55.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:55.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=26297' 2026-02-22T20:02:55.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:55.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:55.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:55.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:55.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:55.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:55.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=26297' 2026-02-22T20:02:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26297 2026-02-22T20:02:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 26297 2026-02-22T20:02:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:55.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:55.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:55.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:55.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:55.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:55.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:55.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:55.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:55.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:55.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:55.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:55.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:55.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:55.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:55.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:55.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:55.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=26500], [object_number=271, tag_tid=23, entry_tid=26499], [object_number=270, tag_tid=23, entry_tid=26498], [object_number=269, tag_tid=23, entry_tid=26497]]], state=connected] 2026-02-22T20:02:56.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:56.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:56.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:56.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:56.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:56.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:56.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:56.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:56.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=26500], [object_number=271, tag_tid=23, entry_tid=26499], [object_number=270, tag_tid=23, entry_tid=26498], [object_number=269, tag_tid=23, entry_tid=26497]]], state=connected] 2026-02-22T20:02:56.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=268, tag_tid=23, entry_tid=26500' 2026-02-22T20:02:56.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=268, tag_tid=23, entry_tid=26500' 2026-02-22T20:02:56.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=26500' '!=' 'object_number=269, tag_tid=23, entry_tid=26297' 2026-02-22T20:02:56.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:56.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=26500' = 'object_number=269, tag_tid=23, entry_tid=26297' 2026-02-22T20:02:56.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=268, tag_tid=23, entry_tid=26500' 2026-02-22T20:02:56.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:56.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:56.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:56.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=26500' 2026-02-22T20:02:56.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:56.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:56.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:56.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:56.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:56.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=26500' 2026-02-22T20:02:56.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:56.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26500 2026-02-22T20:02:56.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 26500 2026-02-22T20:02:56.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:56.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:56.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:56.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:56.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:56.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:56.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:56.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:56.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:56.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:56.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:56.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:56.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:56.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:56.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:56.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:56.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:56.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:56.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:56.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:56.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:56.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:56.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:56.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:56.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:56.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:56.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:56.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:56.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:56.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:56.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:56.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:56.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:56.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=26702], [object_number=269, tag_tid=23, entry_tid=26701], [object_number=268, tag_tid=23, entry_tid=26700], [object_number=271, tag_tid=23, entry_tid=26699]]], state=connected] 2026-02-22T20:02:56.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:56.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:56.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:56.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:56.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:56.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=26702], [object_number=269, tag_tid=23, entry_tid=26701], [object_number=268, tag_tid=23, entry_tid=26700], [object_number=271, tag_tid=23, entry_tid=26699]]], state=connected] 2026-02-22T20:02:56.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=270, tag_tid=23, entry_tid=26702' 2026-02-22T20:02:56.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=270, tag_tid=23, entry_tid=26702' 2026-02-22T20:02:56.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=26702' '!=' 'object_number=268, tag_tid=23, entry_tid=26500' 2026-02-22T20:02:56.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:56.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=26702' = 'object_number=268, tag_tid=23, entry_tid=26500' 2026-02-22T20:02:56.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=270, tag_tid=23, entry_tid=26702' 2026-02-22T20:02:56.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:56.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:56.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:56.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=26702' 2026-02-22T20:02:56.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:56.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:56.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:56.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:56.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=26702' 2026-02-22T20:02:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:56.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26702 2026-02-22T20:02:56.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 26702 2026-02-22T20:02:56.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:56.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:56.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:56.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:56.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:56.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:56.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:56.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:56.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:56.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:56.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:56.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:56.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:56.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:56.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:56.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:56.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:56.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:56.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:56.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=26886], [object_number=269, tag_tid=23, entry_tid=26885], [object_number=268, tag_tid=23, entry_tid=26884], [object_number=271, tag_tid=23, entry_tid=26883]]], state=connected] 2026-02-22T20:02:56.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:56.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:56.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:56.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:56.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:56.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:56.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:56.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:56.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:56.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:56.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:56.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:56.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:56.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:56.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=270, tag_tid=23, entry_tid=26886], [object_number=269, tag_tid=23, entry_tid=26885], [object_number=268, tag_tid=23, entry_tid=26884], [object_number=271, tag_tid=23, entry_tid=26883]]], state=connected] 2026-02-22T20:02:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:56.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=270, tag_tid=23, entry_tid=26886' 2026-02-22T20:02:56.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=270, tag_tid=23, entry_tid=26886' 2026-02-22T20:02:56.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=26886' '!=' 'object_number=270, tag_tid=23, entry_tid=26702' 2026-02-22T20:02:56.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:56.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=270, tag_tid=23, entry_tid=26886' = 'object_number=270, tag_tid=23, entry_tid=26702' 2026-02-22T20:02:56.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=270, tag_tid=23, entry_tid=26886' 2026-02-22T20:02:56.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:56.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:56.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=26886' 2026-02-22T20:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:56.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:56.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:56.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:56.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=270, tag_tid=23, entry_tid=26886' 2026-02-22T20:02:56.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:56.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:56.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26886 2026-02-22T20:02:56.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 26886 2026-02-22T20:02:56.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:56.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:56.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:57.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:57.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:57.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:57.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:57.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:57.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:57.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:57.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:57.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:57.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:57.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:57.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:57.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:57.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:57.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:57.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:57.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=271, tag_tid=23, entry_tid=27099], [object_number=270, tag_tid=23, entry_tid=27098], [object_number=269, tag_tid=23, entry_tid=27097], [object_number=268, tag_tid=23, entry_tid=27096]]], state=connected] 2026-02-22T20:02:57.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:57.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:57.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:57.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:57.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:57.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:57.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:57.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:57.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=271, tag_tid=23, entry_tid=27099], [object_number=270, tag_tid=23, entry_tid=27098], [object_number=269, tag_tid=23, entry_tid=27097], [object_number=268, tag_tid=23, entry_tid=27096]]], state=connected] 2026-02-22T20:02:57.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=271, tag_tid=23, entry_tid=27099' 2026-02-22T20:02:57.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=271, tag_tid=23, entry_tid=27099' 2026-02-22T20:02:57.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=271, tag_tid=23, entry_tid=27099' '!=' 'object_number=270, tag_tid=23, entry_tid=26886' 2026-02-22T20:02:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=271, tag_tid=23, entry_tid=27099' = 'object_number=270, tag_tid=23, entry_tid=26886' 2026-02-22T20:02:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=271, tag_tid=23, entry_tid=27099' 2026-02-22T20:02:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:57.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=271, tag_tid=23, entry_tid=27099' 2026-02-22T20:02:57.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:57.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:57.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:57.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:57.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:57.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=271, tag_tid=23, entry_tid=27099' 2026-02-22T20:02:57.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:57.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27099 2026-02-22T20:02:57.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 27099 2026-02-22T20:02:57.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:57.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:57.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:57.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:57.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:57.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:57.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:57.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:57.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:57.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:57.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:57.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:57.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:57.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:57.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:57.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:57.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:57.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:57.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:57.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:57.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:57.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:57.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:57.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:57.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:57.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:57.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:57.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:57.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:57.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:57.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:57.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:57.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:57.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=27313], [object_number=268, tag_tid=23, entry_tid=27312], [object_number=271, tag_tid=23, entry_tid=27311], [object_number=270, tag_tid=23, entry_tid=27310]]], state=connected] 2026-02-22T20:02:57.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:57.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:57.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:57.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:57.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:57.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:57.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:57.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=269, tag_tid=23, entry_tid=27313], [object_number=268, tag_tid=23, entry_tid=27312], [object_number=271, tag_tid=23, entry_tid=27311], [object_number=270, tag_tid=23, entry_tid=27310]]], state=connected] 2026-02-22T20:02:57.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=269, tag_tid=23, entry_tid=27313' 2026-02-22T20:02:57.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=269, tag_tid=23, entry_tid=27313' 2026-02-22T20:02:57.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=27313' '!=' 'object_number=271, tag_tid=23, entry_tid=27099' 2026-02-22T20:02:57.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:57.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=269, tag_tid=23, entry_tid=27313' = 'object_number=271, tag_tid=23, entry_tid=27099' 2026-02-22T20:02:57.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=269, tag_tid=23, entry_tid=27313' 2026-02-22T20:02:57.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:57.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:57.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:57.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=27313' 2026-02-22T20:02:57.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:57.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:57.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:57.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:57.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:57.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=269, tag_tid=23, entry_tid=27313' 2026-02-22T20:02:57.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:57.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27313 2026-02-22T20:02:57.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 27313 2026-02-22T20:02:57.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:57.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:57.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:57.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:57.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:57.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:57.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:57.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:57.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:57.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:57.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:57.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:57.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:57.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:57.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:57.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:57.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:57.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:57.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:57.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:57.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:57.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:57.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:57.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:57.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:57.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:57.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:57.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:57.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:57.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:57.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:57.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:57.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:57.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:57.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:57.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:57.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:57.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:57.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:57.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:57.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:57.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:57.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:57.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:57.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=27524], [object_number=271, tag_tid=23, entry_tid=27523], [object_number=270, tag_tid=23, entry_tid=27522], [object_number=269, tag_tid=23, entry_tid=27521]]], state=connected] 2026-02-22T20:02:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=268, tag_tid=23, entry_tid=27524], [object_number=271, tag_tid=23, entry_tid=27523], [object_number=270, tag_tid=23, entry_tid=27522], [object_number=269, tag_tid=23, entry_tid=27521]]], state=connected] 2026-02-22T20:02:57.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=268, tag_tid=23, entry_tid=27524' 2026-02-22T20:02:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=268, tag_tid=23, entry_tid=27524' 2026-02-22T20:02:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=27524' '!=' 'object_number=269, tag_tid=23, entry_tid=27313' 2026-02-22T20:02:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=268, tag_tid=23, entry_tid=27524' = 'object_number=269, tag_tid=23, entry_tid=27313' 2026-02-22T20:02:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=268, tag_tid=23, entry_tid=27524' 2026-02-22T20:02:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:57.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:57.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=27524' 2026-02-22T20:02:57.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:57.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:57.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:57.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=268, tag_tid=23, entry_tid=27524' 2026-02-22T20:02:57.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:57.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27524 2026-02-22T20:02:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 27524 2026-02-22T20:02:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:58.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:58.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:58.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:58.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:58.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:58.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:58.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:58.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:58.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:58.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:58.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:58.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:58.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:58.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:58.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:58.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:58.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:58.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:58.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=27701], [object_number=272, tag_tid=23, entry_tid=27700], [object_number=271, tag_tid=23, entry_tid=27699], [object_number=270, tag_tid=23, entry_tid=27698]]], state=connected] 2026-02-22T20:02:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:58.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:58.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:58.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:58.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:58.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:58.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:58.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:58.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 67 2026-02-22T20:02:58.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:58.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:58.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:58.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=27701], [object_number=272, tag_tid=23, entry_tid=27700], [object_number=271, tag_tid=23, entry_tid=27699], [object_number=270, tag_tid=23, entry_tid=27698]]], state=connected] 2026-02-22T20:02:58.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=273, tag_tid=23, entry_tid=27701' 2026-02-22T20:02:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=273, tag_tid=23, entry_tid=27701' 2026-02-22T20:02:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=27701' '!=' 'object_number=268, tag_tid=23, entry_tid=27524' 2026-02-22T20:02:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=27701' = 'object_number=268, tag_tid=23, entry_tid=27524' 2026-02-22T20:02:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=273, tag_tid=23, entry_tid=27701' 2026-02-22T20:02:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:58.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=27701' 2026-02-22T20:02:58.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:58.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:58.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:58.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:58.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:58.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:58.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:58.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=27701' 2026-02-22T20:02:58.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:58.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27701 2026-02-22T20:02:58.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 27701 2026-02-22T20:02:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:58.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:58.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:58.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:58.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:58.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:58.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:58.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:58.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:58.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:58.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:58.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:58.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:58.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:58.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:58.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:58.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:58.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:58.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:58.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:58.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:58.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:02:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:58.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=274, tag_tid=23, entry_tid=27866], [object_number=273, tag_tid=23, entry_tid=27865], [object_number=272, tag_tid=23, entry_tid=27864], [object_number=275, tag_tid=23, entry_tid=27863]]], state=connected] 2026-02-22T20:02:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:58.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:58.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:58.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:58.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:58.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:02:58.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:58.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:58.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:58.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=274, tag_tid=23, entry_tid=27866], [object_number=273, tag_tid=23, entry_tid=27865], [object_number=272, tag_tid=23, entry_tid=27864], [object_number=275, tag_tid=23, entry_tid=27863]]], state=connected] 2026-02-22T20:02:58.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=274, tag_tid=23, entry_tid=27866' 2026-02-22T20:02:58.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=274, tag_tid=23, entry_tid=27866' 2026-02-22T20:02:58.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=274, tag_tid=23, entry_tid=27866' '!=' 'object_number=273, tag_tid=23, entry_tid=27701' 2026-02-22T20:02:58.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:58.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=274, tag_tid=23, entry_tid=27866' = 'object_number=273, tag_tid=23, entry_tid=27701' 2026-02-22T20:02:58.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=274, tag_tid=23, entry_tid=27866' 2026-02-22T20:02:58.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:58.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:58.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:58.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:58.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=274, tag_tid=23, entry_tid=27866' 2026-02-22T20:02:58.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:58.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:58.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:58.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:58.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:58.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:58.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:58.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=274, tag_tid=23, entry_tid=27866' 2026-02-22T20:02:58.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:58.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:58.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27866 2026-02-22T20:02:58.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 27866 2026-02-22T20:02:58.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:58.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:58.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:58.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:58.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:58.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:58.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:58.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:58.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:02:58.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:58.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:58.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:58.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=28073], [object_number=272, tag_tid=23, entry_tid=28072], [object_number=275, tag_tid=23, entry_tid=28071], [object_number=274, tag_tid=23, entry_tid=28070]]], state=connected] 2026-02-22T20:02:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:58.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:58.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:58.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:58.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:58.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:58.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:58.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:58.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:58.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:58.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:58.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:58.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:02:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=28073], [object_number=272, tag_tid=23, entry_tid=28072], [object_number=275, tag_tid=23, entry_tid=28071], [object_number=274, tag_tid=23, entry_tid=28070]]], state=connected] 2026-02-22T20:02:59.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=273, tag_tid=23, entry_tid=28073' 2026-02-22T20:02:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=273, tag_tid=23, entry_tid=28073' 2026-02-22T20:02:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=28073' '!=' 'object_number=274, tag_tid=23, entry_tid=27866' 2026-02-22T20:02:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:59.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=28073' = 'object_number=274, tag_tid=23, entry_tid=27866' 2026-02-22T20:02:59.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=273, tag_tid=23, entry_tid=28073' 2026-02-22T20:02:59.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:59.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:59.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:59.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=28073' 2026-02-22T20:02:59.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:59.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:59.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:59.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:59.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:59.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=28073' 2026-02-22T20:02:59.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:59.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28073 2026-02-22T20:02:59.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 28073 2026-02-22T20:02:59.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:59.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:59.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:59.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:59.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:59.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:59.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:59.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:59.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:59.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:59.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:59.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:59.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:59.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:59.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:59.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:59.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:59.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:59.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:59.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:59.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:59.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:59.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:59.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:59.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:59.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:59.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:59.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:59.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:59.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:59.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:59.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:59.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:59.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:02:59.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:59.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:59.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:59.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=28288], [object_number=275, tag_tid=23, entry_tid=28287], [object_number=274, tag_tid=23, entry_tid=28286], [object_number=273, tag_tid=23, entry_tid=28285]]], state=connected] 2026-02-22T20:02:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:59.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:59.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:59.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:02:59.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:59.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:59.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:59.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=28288], [object_number=275, tag_tid=23, entry_tid=28287], [object_number=274, tag_tid=23, entry_tid=28286], [object_number=273, tag_tid=23, entry_tid=28285]]], state=connected] 2026-02-22T20:02:59.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=272, tag_tid=23, entry_tid=28288' 2026-02-22T20:02:59.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=272, tag_tid=23, entry_tid=28288' 2026-02-22T20:02:59.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=28288' '!=' 'object_number=273, tag_tid=23, entry_tid=28073' 2026-02-22T20:02:59.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:59.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=28288' = 'object_number=273, tag_tid=23, entry_tid=28073' 2026-02-22T20:02:59.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=272, tag_tid=23, entry_tid=28288' 2026-02-22T20:02:59.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:59.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:59.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:59.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=28288' 2026-02-22T20:02:59.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:59.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:59.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:59.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:59.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:59.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=28288' 2026-02-22T20:02:59.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:59.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28288 2026-02-22T20:02:59.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 28288 2026-02-22T20:02:59.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:59.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:59.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:59.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:59.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:59.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:59.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:59.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:59.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:59.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:59.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:59.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:59.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:59.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:02:59.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:02:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:02:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:02:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:02:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:59.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:02:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=275, tag_tid=23, entry_tid=28483], [object_number=274, tag_tid=23, entry_tid=28482], [object_number=273, tag_tid=23, entry_tid=28481], [object_number=272, tag_tid=23, entry_tid=28480]]], state=connected] 2026-02-22T20:02:59.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:59.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:02:59.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:59.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:59.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:59.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:02:59.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:02:59.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:02:59.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:02:59.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:02:59.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:59.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:02:59.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:02:59.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:02:59.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:02:59.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:02:59.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:02:59.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=275, tag_tid=23, entry_tid=28483], [object_number=274, tag_tid=23, entry_tid=28482], [object_number=273, tag_tid=23, entry_tid=28481], [object_number=272, tag_tid=23, entry_tid=28480]]], state=connected] 2026-02-22T20:02:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:02:59.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=275, tag_tid=23, entry_tid=28483' 2026-02-22T20:02:59.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=275, tag_tid=23, entry_tid=28483' 2026-02-22T20:02:59.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=275, tag_tid=23, entry_tid=28483' '!=' 'object_number=272, tag_tid=23, entry_tid=28288' 2026-02-22T20:02:59.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:02:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=275, tag_tid=23, entry_tid=28483' = 'object_number=272, tag_tid=23, entry_tid=28288' 2026-02-22T20:02:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=275, tag_tid=23, entry_tid=28483' 2026-02-22T20:02:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:02:59.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=275, tag_tid=23, entry_tid=28483' 2026-02-22T20:02:59.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:02:59.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:02:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:02:59.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:59.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:02:59.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=275, tag_tid=23, entry_tid=28483' 2026-02-22T20:02:59.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:02:59.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:02:59.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28483 2026-02-22T20:02:59.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 28483 2026-02-22T20:02:59.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:02:59.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:59.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:02:59.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:02:59.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:02:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:02:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:02:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:02:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:02:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:02:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:02:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:02:59.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:02:59.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:02:59.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:59.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:02:59.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:59.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:02:59.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:02:59.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:02:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:02:59.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:02:59.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:02:59.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:59.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:02:59.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:02:59.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:02:59.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:02:59.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:59.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:00.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:00.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:00.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:00.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:00.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:00.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:00.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:00.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:00.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=275, tag_tid=23, entry_tid=28703], [object_number=274, tag_tid=23, entry_tid=28702], [object_number=273, tag_tid=23, entry_tid=28701], [object_number=272, tag_tid=23, entry_tid=28700]]], state=connected] 2026-02-22T20:03:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:00.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:00.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:00.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:00.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=275, tag_tid=23, entry_tid=28703], [object_number=274, tag_tid=23, entry_tid=28702], [object_number=273, tag_tid=23, entry_tid=28701], [object_number=272, tag_tid=23, entry_tid=28700]]], state=connected] 2026-02-22T20:03:00.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=275, tag_tid=23, entry_tid=28703' 2026-02-22T20:03:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=275, tag_tid=23, entry_tid=28703' 2026-02-22T20:03:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=275, tag_tid=23, entry_tid=28703' '!=' 'object_number=275, tag_tid=23, entry_tid=28483' 2026-02-22T20:03:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=275, tag_tid=23, entry_tid=28703' = 'object_number=275, tag_tid=23, entry_tid=28483' 2026-02-22T20:03:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=275, tag_tid=23, entry_tid=28703' 2026-02-22T20:03:00.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:00.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:00.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:00.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=275, tag_tid=23, entry_tid=28703' 2026-02-22T20:03:00.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:00.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:00.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=275, tag_tid=23, entry_tid=28703' 2026-02-22T20:03:00.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:00.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28703 2026-02-22T20:03:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 28703 2026-02-22T20:03:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:00.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:00.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:00.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:00.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:00.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:00.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:00.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:00.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:00.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:00.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:00.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:00.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:00.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:00.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=28916], [object_number=275, tag_tid=23, entry_tid=28915], [object_number=274, tag_tid=23, entry_tid=28914], [object_number=273, tag_tid=23, entry_tid=28913]]], state=connected] 2026-02-22T20:03:00.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:00.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:00.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:00.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:00.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:00.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:00.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=28916], [object_number=275, tag_tid=23, entry_tid=28915], [object_number=274, tag_tid=23, entry_tid=28914], [object_number=273, tag_tid=23, entry_tid=28913]]], state=connected] 2026-02-22T20:03:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=272, tag_tid=23, entry_tid=28916' 2026-02-22T20:03:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=272, tag_tid=23, entry_tid=28916' 2026-02-22T20:03:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=28916' '!=' 'object_number=275, tag_tid=23, entry_tid=28703' 2026-02-22T20:03:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=28916' = 'object_number=275, tag_tid=23, entry_tid=28703' 2026-02-22T20:03:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=272, tag_tid=23, entry_tid=28916' 2026-02-22T20:03:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:00.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=28916' 2026-02-22T20:03:00.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:00.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:00.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=28916' 2026-02-22T20:03:00.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:00.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28916 2026-02-22T20:03:00.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 28916 2026-02-22T20:03:00.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:00.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:00.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:00.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:00.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:00.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:00.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:00.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:00.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:00.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:00.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=275, tag_tid=23, entry_tid=29119], [object_number=274, tag_tid=23, entry_tid=29118], [object_number=273, tag_tid=23, entry_tid=29117], [object_number=272, tag_tid=23, entry_tid=29116]]], state=connected] 2026-02-22T20:03:00.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:00.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:00.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:00.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:00.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=275, tag_tid=23, entry_tid=29119], [object_number=274, tag_tid=23, entry_tid=29118], [object_number=273, tag_tid=23, entry_tid=29117], [object_number=272, tag_tid=23, entry_tid=29116]]], state=connected] 2026-02-22T20:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=275, tag_tid=23, entry_tid=29119' 2026-02-22T20:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=275, tag_tid=23, entry_tid=29119' 2026-02-22T20:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=275, tag_tid=23, entry_tid=29119' '!=' 'object_number=272, tag_tid=23, entry_tid=28916' 2026-02-22T20:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=275, tag_tid=23, entry_tid=29119' = 'object_number=272, tag_tid=23, entry_tid=28916' 2026-02-22T20:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=275, tag_tid=23, entry_tid=29119' 2026-02-22T20:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:00.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:00.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=275, tag_tid=23, entry_tid=29119' 2026-02-22T20:03:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:00.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:00.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:00.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=275, tag_tid=23, entry_tid=29119' 2026-02-22T20:03:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29119 2026-02-22T20:03:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 29119 2026-02-22T20:03:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:01.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:01.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:01.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:01.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:01.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:01.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:01.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:01.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:01.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:01.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:01.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:01.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=29317], [object_number=272, tag_tid=23, entry_tid=29316], [object_number=275, tag_tid=23, entry_tid=29315], [object_number=274, tag_tid=23, entry_tid=29314]]], state=connected] 2026-02-22T20:03:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:01.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:01.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:01.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:01.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:01.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:01.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=29317], [object_number=272, tag_tid=23, entry_tid=29316], [object_number=275, tag_tid=23, entry_tid=29315], [object_number=274, tag_tid=23, entry_tid=29314]]], state=connected] 2026-02-22T20:03:01.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=273, tag_tid=23, entry_tid=29317' 2026-02-22T20:03:01.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=273, tag_tid=23, entry_tid=29317' 2026-02-22T20:03:01.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=29317' '!=' 'object_number=275, tag_tid=23, entry_tid=29119' 2026-02-22T20:03:01.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:01.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=29317' = 'object_number=275, tag_tid=23, entry_tid=29119' 2026-02-22T20:03:01.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=273, tag_tid=23, entry_tid=29317' 2026-02-22T20:03:01.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:01.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:01.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:01.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=29317' 2026-02-22T20:03:01.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:01.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:01.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:01.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:01.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=29317' 2026-02-22T20:03:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29317 2026-02-22T20:03:01.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 29317 2026-02-22T20:03:01.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:01.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:01.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:01.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:01.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:01.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:01.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:01.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:01.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:01.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:01.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:01.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:01.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=29525], [object_number=272, tag_tid=23, entry_tid=29524], [object_number=275, tag_tid=23, entry_tid=29523], [object_number=274, tag_tid=23, entry_tid=29522]]], state=connected] 2026-02-22T20:03:01.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:01.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:01.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:01.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:01.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:01.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:01.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=29525], [object_number=272, tag_tid=23, entry_tid=29524], [object_number=275, tag_tid=23, entry_tid=29523], [object_number=274, tag_tid=23, entry_tid=29522]]], state=connected] 2026-02-22T20:03:01.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=273, tag_tid=23, entry_tid=29525' 2026-02-22T20:03:01.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=273, tag_tid=23, entry_tid=29525' 2026-02-22T20:03:01.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=29525' '!=' 'object_number=273, tag_tid=23, entry_tid=29317' 2026-02-22T20:03:01.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:01.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=29525' = 'object_number=273, tag_tid=23, entry_tid=29317' 2026-02-22T20:03:01.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=273, tag_tid=23, entry_tid=29525' 2026-02-22T20:03:01.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:01.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:01.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:01.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=29525' 2026-02-22T20:03:01.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:01.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:01.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:01.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:01.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:01.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=29525' 2026-02-22T20:03:01.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:01.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29525 2026-02-22T20:03:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 29525 2026-02-22T20:03:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:01.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:01.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:01.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:01.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:01.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:01.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:01.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:01.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:01.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:01.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:01.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:01.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:01.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:01.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:01.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:01.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=274, tag_tid=23, entry_tid=29738], [object_number=273, tag_tid=23, entry_tid=29737], [object_number=272, tag_tid=23, entry_tid=29736], [object_number=275, tag_tid=23, entry_tid=29735]]], state=connected] 2026-02-22T20:03:01.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=274, tag_tid=23, entry_tid=29738], [object_number=273, tag_tid=23, entry_tid=29737], [object_number=272, tag_tid=23, entry_tid=29736], [object_number=275, tag_tid=23, entry_tid=29735]]], state=connected] 2026-02-22T20:03:01.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:01.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=274, tag_tid=23, entry_tid=29738' 2026-02-22T20:03:01.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=274, tag_tid=23, entry_tid=29738' 2026-02-22T20:03:01.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=274, tag_tid=23, entry_tid=29738' '!=' 'object_number=273, tag_tid=23, entry_tid=29525' 2026-02-22T20:03:01.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:01.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=274, tag_tid=23, entry_tid=29738' = 'object_number=273, tag_tid=23, entry_tid=29525' 2026-02-22T20:03:01.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=274, tag_tid=23, entry_tid=29738' 2026-02-22T20:03:01.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:01.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:01.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=274, tag_tid=23, entry_tid=29738' 2026-02-22T20:03:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:01.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:01.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:01.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:01.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:01.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=274, tag_tid=23, entry_tid=29738' 2026-02-22T20:03:01.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:01.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:01.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29738 2026-02-22T20:03:01.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 29738 2026-02-22T20:03:01.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:01.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:01.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:02.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:02.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:02.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:02.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:02.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:02.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:02.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:02.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:02.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:02.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:02.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:02.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:02.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:02.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:02.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:02.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:02.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:02.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:02.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:02.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:02.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:02.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:02.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=29952], [object_number=275, tag_tid=23, entry_tid=29951], [object_number=274, tag_tid=23, entry_tid=29950], [object_number=273, tag_tid=23, entry_tid=29949]]], state=connected] 2026-02-22T20:03:02.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:02.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:02.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:02.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:02.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=29952], [object_number=275, tag_tid=23, entry_tid=29951], [object_number=274, tag_tid=23, entry_tid=29950], [object_number=273, tag_tid=23, entry_tid=29949]]], state=connected] 2026-02-22T20:03:02.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:02.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=272, tag_tid=23, entry_tid=29952' 2026-02-22T20:03:02.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=272, tag_tid=23, entry_tid=29952' 2026-02-22T20:03:02.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=29952' '!=' 'object_number=274, tag_tid=23, entry_tid=29738' 2026-02-22T20:03:02.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:02.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=29952' = 'object_number=274, tag_tid=23, entry_tid=29738' 2026-02-22T20:03:02.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=272, tag_tid=23, entry_tid=29952' 2026-02-22T20:03:02.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=29952' 2026-02-22T20:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:02.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:02.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:02.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:02.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:02.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:02.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=29952' 2026-02-22T20:03:02.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:02.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29952 2026-02-22T20:03:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 29952 2026-02-22T20:03:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:03.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:03.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:03.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:03.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:03.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:03.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:03.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:03.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:03.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=275, tag_tid=23, entry_tid=30251], [object_number=274, tag_tid=23, entry_tid=30250], [object_number=273, tag_tid=23, entry_tid=30249], [object_number=272, tag_tid=23, entry_tid=30248]]], state=connected] 2026-02-22T20:03:03.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:03.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:03.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:03.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:03.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:03.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:03.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:03.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:03.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:03.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:03.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=275, tag_tid=23, entry_tid=30251], [object_number=274, tag_tid=23, entry_tid=30250], [object_number=273, tag_tid=23, entry_tid=30249], [object_number=272, tag_tid=23, entry_tid=30248]]], state=connected] 2026-02-22T20:03:03.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=275, tag_tid=23, entry_tid=30251' 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=275, tag_tid=23, entry_tid=30251' 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=275, tag_tid=23, entry_tid=30251' '!=' 'object_number=272, tag_tid=23, entry_tid=29952' 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=275, tag_tid=23, entry_tid=30251' = 'object_number=272, tag_tid=23, entry_tid=29952' 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=275, tag_tid=23, entry_tid=30251' 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:03.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=275, tag_tid=23, entry_tid=30251' 2026-02-22T20:03:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:03.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:03.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=275, tag_tid=23, entry_tid=30251' 2026-02-22T20:03:03.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:03.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30251 2026-02-22T20:03:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 30251 2026-02-22T20:03:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:03.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:03.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:03.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:03.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:03.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:03.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:03.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:03.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:03.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:03.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=274, tag_tid=23, entry_tid=30442], [object_number=273, tag_tid=23, entry_tid=30441], [object_number=272, tag_tid=23, entry_tid=30440], [object_number=275, tag_tid=23, entry_tid=30439]]], state=connected] 2026-02-22T20:03:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:03.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:03.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:03.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:03.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:03.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:03.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:03.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:03.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:03.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=274, tag_tid=23, entry_tid=30442], [object_number=273, tag_tid=23, entry_tid=30441], [object_number=272, tag_tid=23, entry_tid=30440], [object_number=275, tag_tid=23, entry_tid=30439]]], state=connected] 2026-02-22T20:03:03.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=274, tag_tid=23, entry_tid=30442' 2026-02-22T20:03:03.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=274, tag_tid=23, entry_tid=30442' 2026-02-22T20:03:03.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=274, tag_tid=23, entry_tid=30442' '!=' 'object_number=275, tag_tid=23, entry_tid=30251' 2026-02-22T20:03:03.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:03.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=274, tag_tid=23, entry_tid=30442' = 'object_number=275, tag_tid=23, entry_tid=30251' 2026-02-22T20:03:03.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=274, tag_tid=23, entry_tid=30442' 2026-02-22T20:03:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=274, tag_tid=23, entry_tid=30442' 2026-02-22T20:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:03.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:03.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:03.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:03.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=274, tag_tid=23, entry_tid=30442' 2026-02-22T20:03:03.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:03.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30442 2026-02-22T20:03:03.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 30442 2026-02-22T20:03:03.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:03.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:03.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:03.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:03.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:03.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:03.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:03.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:03.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:03.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:03.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:03.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:03.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:03.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:03.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:03.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:03.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:03.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:03.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=30640], [object_number=275, tag_tid=23, entry_tid=30639], [object_number=274, tag_tid=23, entry_tid=30638], [object_number=273, tag_tid=23, entry_tid=30637]]], state=connected] 2026-02-22T20:03:03.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:03.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:03.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:03.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:03.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=30640], [object_number=275, tag_tid=23, entry_tid=30639], [object_number=274, tag_tid=23, entry_tid=30638], [object_number=273, tag_tid=23, entry_tid=30637]]], state=connected] 2026-02-22T20:03:03.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:03.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=272, tag_tid=23, entry_tid=30640' 2026-02-22T20:03:03.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=272, tag_tid=23, entry_tid=30640' 2026-02-22T20:03:03.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=30640' '!=' 'object_number=274, tag_tid=23, entry_tid=30442' 2026-02-22T20:03:03.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:03.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=30640' = 'object_number=274, tag_tid=23, entry_tid=30442' 2026-02-22T20:03:03.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=272, tag_tid=23, entry_tid=30640' 2026-02-22T20:03:03.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:03.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:03.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:03.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=30640' 2026-02-22T20:03:03.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:03.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:03.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:03.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:03.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:03.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=30640' 2026-02-22T20:03:03.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:03.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:03.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30640 2026-02-22T20:03:03.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 30640 2026-02-22T20:03:03.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:03.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:03.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:04.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:04.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:04.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:04.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:04.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:04.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:04.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:04.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:04.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:04.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:04.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:04.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:04.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:04.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:04.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:04.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:04.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:04.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:04.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=30857], [object_number=272, tag_tid=23, entry_tid=30856], [object_number=275, tag_tid=23, entry_tid=30855], [object_number=274, tag_tid=23, entry_tid=30854]]], state=connected] 2026-02-22T20:03:04.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:04.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:04.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:04.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:04.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:04.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:04.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:04.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:04.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:04.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=30857], [object_number=272, tag_tid=23, entry_tid=30856], [object_number=275, tag_tid=23, entry_tid=30855], [object_number=274, tag_tid=23, entry_tid=30854]]], state=connected] 2026-02-22T20:03:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=273, tag_tid=23, entry_tid=30857' 2026-02-22T20:03:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=273, tag_tid=23, entry_tid=30857' 2026-02-22T20:03:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=30857' '!=' 'object_number=272, tag_tid=23, entry_tid=30640' 2026-02-22T20:03:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=30857' = 'object_number=272, tag_tid=23, entry_tid=30640' 2026-02-22T20:03:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=273, tag_tid=23, entry_tid=30857' 2026-02-22T20:03:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:04.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=30857' 2026-02-22T20:03:04.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:04.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:04.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:04.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:04.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:04.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=30857' 2026-02-22T20:03:04.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:04.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30857 2026-02-22T20:03:04.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 30857 2026-02-22T20:03:04.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:04.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:04.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:04.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:04.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:04.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:04.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:04.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:04.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:04.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:04.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:04.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:04.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:04.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:04.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:04.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:04.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:04.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:04.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:04.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:04.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=31060], [object_number=275, tag_tid=23, entry_tid=31059], [object_number=274, tag_tid=23, entry_tid=31058], [object_number=273, tag_tid=23, entry_tid=31057]]], state=connected] 2026-02-22T20:03:04.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:04.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:04.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:04.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:04.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:04.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:04.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:04.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:04.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:04.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=31060], [object_number=275, tag_tid=23, entry_tid=31059], [object_number=274, tag_tid=23, entry_tid=31058], [object_number=273, tag_tid=23, entry_tid=31057]]], state=connected] 2026-02-22T20:03:04.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=272, tag_tid=23, entry_tid=31060' 2026-02-22T20:03:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=272, tag_tid=23, entry_tid=31060' 2026-02-22T20:03:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=31060' '!=' 'object_number=273, tag_tid=23, entry_tid=30857' 2026-02-22T20:03:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=31060' = 'object_number=273, tag_tid=23, entry_tid=30857' 2026-02-22T20:03:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=272, tag_tid=23, entry_tid=31060' 2026-02-22T20:03:04.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:04.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:04.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:04.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=31060' 2026-02-22T20:03:04.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:04.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:04.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:04.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:04.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=31060' 2026-02-22T20:03:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31060 2026-02-22T20:03:04.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 31060 2026-02-22T20:03:04.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:04.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:04.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:04.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:04.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:04.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:04.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:04.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:04.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:04.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:04.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:04.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:04.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:04.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:04.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:04.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:04.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:04.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:04.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:04.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:04.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:04.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:04.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:04.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:04.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:04.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:04.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:04.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:04.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:04.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:04.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:04.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:04.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:04.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=31273], [object_number=272, tag_tid=23, entry_tid=31272], [object_number=275, tag_tid=23, entry_tid=31271], [object_number=274, tag_tid=23, entry_tid=31270]]], state=connected] 2026-02-22T20:03:04.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:04.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:04.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:04.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:04.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:04.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:04.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:04.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:04.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:04.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=31273], [object_number=272, tag_tid=23, entry_tid=31272], [object_number=275, tag_tid=23, entry_tid=31271], [object_number=274, tag_tid=23, entry_tid=31270]]], state=connected] 2026-02-22T20:03:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=273, tag_tid=23, entry_tid=31273' 2026-02-22T20:03:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=273, tag_tid=23, entry_tid=31273' 2026-02-22T20:03:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=31273' '!=' 'object_number=272, tag_tid=23, entry_tid=31060' 2026-02-22T20:03:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=31273' = 'object_number=272, tag_tid=23, entry_tid=31060' 2026-02-22T20:03:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=273, tag_tid=23, entry_tid=31273' 2026-02-22T20:03:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=31273' 2026-02-22T20:03:04.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:04.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:04.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:04.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:04.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:04.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=31273' 2026-02-22T20:03:04.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:04.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:04.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31273 2026-02-22T20:03:04.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 31273 2026-02-22T20:03:04.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:04.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:04.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:05.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=31492], [object_number=275, tag_tid=23, entry_tid=31491], [object_number=274, tag_tid=23, entry_tid=31490], [object_number=273, tag_tid=23, entry_tid=31489]]], state=connected] 2026-02-22T20:03:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:05.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:05.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:05.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:05.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=272, tag_tid=23, entry_tid=31492], [object_number=275, tag_tid=23, entry_tid=31491], [object_number=274, tag_tid=23, entry_tid=31490], [object_number=273, tag_tid=23, entry_tid=31489]]], state=connected] 2026-02-22T20:03:05.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=272, tag_tid=23, entry_tid=31492' 2026-02-22T20:03:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=272, tag_tid=23, entry_tid=31492' 2026-02-22T20:03:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=31492' '!=' 'object_number=273, tag_tid=23, entry_tid=31273' 2026-02-22T20:03:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=272, tag_tid=23, entry_tid=31492' = 'object_number=273, tag_tid=23, entry_tid=31273' 2026-02-22T20:03:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=272, tag_tid=23, entry_tid=31492' 2026-02-22T20:03:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:05.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:05.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=31492' 2026-02-22T20:03:05.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:05.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:05.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:05.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:05.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:05.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:05.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:05.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=272, tag_tid=23, entry_tid=31492' 2026-02-22T20:03:05.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:05.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:05.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31492 2026-02-22T20:03:05.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 31492 2026-02-22T20:03:05.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:05.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:05.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:05.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:05.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:05.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:05.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:05.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:05.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=31685], [object_number=272, tag_tid=23, entry_tid=31684], [object_number=275, tag_tid=23, entry_tid=31683], [object_number=274, tag_tid=23, entry_tid=31682]]], state=connected] 2026-02-22T20:03:05.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:05.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:05.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:05.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:05.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:05.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:05.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:05.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:05.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:05.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:05.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=31685], [object_number=272, tag_tid=23, entry_tid=31684], [object_number=275, tag_tid=23, entry_tid=31683], [object_number=274, tag_tid=23, entry_tid=31682]]], state=connected] 2026-02-22T20:03:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=273, tag_tid=23, entry_tid=31685' 2026-02-22T20:03:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=273, tag_tid=23, entry_tid=31685' 2026-02-22T20:03:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=31685' '!=' 'object_number=272, tag_tid=23, entry_tid=31492' 2026-02-22T20:03:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=31685' = 'object_number=272, tag_tid=23, entry_tid=31492' 2026-02-22T20:03:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=273, tag_tid=23, entry_tid=31685' 2026-02-22T20:03:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:05.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:05.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=31685' 2026-02-22T20:03:05.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:05.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:05.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:05.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:05.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:05.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:05.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:05.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=31685' 2026-02-22T20:03:05.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:05.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:05.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31685 2026-02-22T20:03:05.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 31685 2026-02-22T20:03:05.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:05.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:05.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:05.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:05.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:05.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:05.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:05.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:05.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:05.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:05.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:05.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:05.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:05.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:05.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:05.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:05.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:05.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:05.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:05.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:05.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:06.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=31897], [object_number=272, tag_tid=23, entry_tid=31896], [object_number=275, tag_tid=23, entry_tid=31895], [object_number=274, tag_tid=23, entry_tid=31894]]], state=connected] 2026-02-22T20:03:06.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:06.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:06.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:06.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:06.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:06.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 68 2026-02-22T20:03:06.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:06.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:06.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:06.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=273, tag_tid=23, entry_tid=31897], [object_number=272, tag_tid=23, entry_tid=31896], [object_number=275, tag_tid=23, entry_tid=31895], [object_number=274, tag_tid=23, entry_tid=31894]]], state=connected] 2026-02-22T20:03:06.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=273, tag_tid=23, entry_tid=31897' 2026-02-22T20:03:06.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=273, tag_tid=23, entry_tid=31897' 2026-02-22T20:03:06.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=31897' '!=' 'object_number=273, tag_tid=23, entry_tid=31685' 2026-02-22T20:03:06.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:06.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=273, tag_tid=23, entry_tid=31897' = 'object_number=273, tag_tid=23, entry_tid=31685' 2026-02-22T20:03:06.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=273, tag_tid=23, entry_tid=31897' 2026-02-22T20:03:06.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:06.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:06.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:06.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=31897' 2026-02-22T20:03:06.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:06.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:06.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:06.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:06.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:06.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=273, tag_tid=23, entry_tid=31897' 2026-02-22T20:03:06.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:06.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=31897 2026-02-22T20:03:06.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 31897 2026-02-22T20:03:06.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:06.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:06.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:06.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:06.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=32065], [object_number=276, tag_tid=23, entry_tid=32064], [object_number=279, tag_tid=23, entry_tid=32063], [object_number=278, tag_tid=23, entry_tid=32062]]], state=connected] 2026-02-22T20:03:06.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:06.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:06.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:06.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:06.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:06.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:06.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:06.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:06.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:06.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:06.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=32065], [object_number=276, tag_tid=23, entry_tid=32064], [object_number=279, tag_tid=23, entry_tid=32063], [object_number=278, tag_tid=23, entry_tid=32062]]], state=connected] 2026-02-22T20:03:06.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=277, tag_tid=23, entry_tid=32065' 2026-02-22T20:03:06.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=277, tag_tid=23, entry_tid=32065' 2026-02-22T20:03:06.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=32065' '!=' 'object_number=273, tag_tid=23, entry_tid=31897' 2026-02-22T20:03:06.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:06.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=32065' = 'object_number=273, tag_tid=23, entry_tid=31897' 2026-02-22T20:03:06.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=277, tag_tid=23, entry_tid=32065' 2026-02-22T20:03:06.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:06.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:06.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:06.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=32065' 2026-02-22T20:03:06.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:06.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:06.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:06.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:06.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=32065' 2026-02-22T20:03:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=32065 2026-02-22T20:03:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 32065 2026-02-22T20:03:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:06.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:06.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:06.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:06.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:06.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:06.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:06.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:06.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:06.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:06.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=279, tag_tid=23, entry_tid=32283], [object_number=278, tag_tid=23, entry_tid=32282], [object_number=277, tag_tid=23, entry_tid=32281], [object_number=276, tag_tid=23, entry_tid=32280]]], state=connected] 2026-02-22T20:03:06.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=279, tag_tid=23, entry_tid=32283], [object_number=278, tag_tid=23, entry_tid=32282], [object_number=277, tag_tid=23, entry_tid=32281], [object_number=276, tag_tid=23, entry_tid=32280]]], state=connected] 2026-02-22T20:03:06.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=279, tag_tid=23, entry_tid=32283' 2026-02-22T20:03:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=279, tag_tid=23, entry_tid=32283' 2026-02-22T20:03:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=279, tag_tid=23, entry_tid=32283' '!=' 'object_number=277, tag_tid=23, entry_tid=32065' 2026-02-22T20:03:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=279, tag_tid=23, entry_tid=32283' = 'object_number=277, tag_tid=23, entry_tid=32065' 2026-02-22T20:03:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=279, tag_tid=23, entry_tid=32283' 2026-02-22T20:03:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=279, tag_tid=23, entry_tid=32283' 2026-02-22T20:03:06.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:06.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:06.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:06.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:06.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:06.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=279, tag_tid=23, entry_tid=32283' 2026-02-22T20:03:06.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:06.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:06.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=32283 2026-02-22T20:03:06.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 32283 2026-02-22T20:03:06.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:06.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:06.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:07.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:07.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:07.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:07.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:07.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:07.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:07.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:07.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:07.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:07.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:07.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:07.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:07.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:07.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:07.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:07.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:07.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:07.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:07.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=32501], [object_number=276, tag_tid=23, entry_tid=32500], [object_number=279, tag_tid=23, entry_tid=32499], [object_number=278, tag_tid=23, entry_tid=32498]]], state=connected] 2026-02-22T20:03:07.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:07.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:07.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:07.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:07.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:07.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:07.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:07.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:07.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=32501], [object_number=276, tag_tid=23, entry_tid=32500], [object_number=279, tag_tid=23, entry_tid=32499], [object_number=278, tag_tid=23, entry_tid=32498]]], state=connected] 2026-02-22T20:03:07.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=277, tag_tid=23, entry_tid=32501' 2026-02-22T20:03:07.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=277, tag_tid=23, entry_tid=32501' 2026-02-22T20:03:07.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=32501' '!=' 'object_number=279, tag_tid=23, entry_tid=32283' 2026-02-22T20:03:07.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:07.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=32501' = 'object_number=279, tag_tid=23, entry_tid=32283' 2026-02-22T20:03:07.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=277, tag_tid=23, entry_tid=32501' 2026-02-22T20:03:07.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:07.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:07.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:07.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=32501' 2026-02-22T20:03:07.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:07.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:07.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:07.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:07.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:07.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=32501' 2026-02-22T20:03:07.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:07.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=32501 2026-02-22T20:03:07.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 32501 2026-02-22T20:03:07.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:07.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:07.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:07.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:07.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:07.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:07.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:07.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:07.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:07.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:07.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:07.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:07.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:07.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:07.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:07.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:07.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:07.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:07.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:07.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=32720], [object_number=279, tag_tid=23, entry_tid=32719], [object_number=278, tag_tid=23, entry_tid=32718], [object_number=277, tag_tid=23, entry_tid=32717]]], state=connected] 2026-02-22T20:03:07.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:07.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:07.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:07.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:07.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:07.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=32720], [object_number=279, tag_tid=23, entry_tid=32719], [object_number=278, tag_tid=23, entry_tid=32718], [object_number=277, tag_tid=23, entry_tid=32717]]], state=connected] 2026-02-22T20:03:07.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=276, tag_tid=23, entry_tid=32720' 2026-02-22T20:03:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=276, tag_tid=23, entry_tid=32720' 2026-02-22T20:03:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=32720' '!=' 'object_number=277, tag_tid=23, entry_tid=32501' 2026-02-22T20:03:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:07.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=32720' = 'object_number=277, tag_tid=23, entry_tid=32501' 2026-02-22T20:03:07.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=276, tag_tid=23, entry_tid=32720' 2026-02-22T20:03:07.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:07.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:07.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:07.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=32720' 2026-02-22T20:03:07.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:07.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:07.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:07.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:07.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:07.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=32720' 2026-02-22T20:03:07.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:07.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=32720 2026-02-22T20:03:07.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 32720 2026-02-22T20:03:07.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:07.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:07.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:07.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:07.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:07.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:07.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:07.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:07.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:07.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:07.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:07.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:07.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:07.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:07.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:07.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=278, tag_tid=23, entry_tid=32934], [object_number=277, tag_tid=23, entry_tid=32933], [object_number=276, tag_tid=23, entry_tid=32932], [object_number=279, tag_tid=23, entry_tid=32931]]], state=connected] 2026-02-22T20:03:07.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:07.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:07.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:07.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:07.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:07.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:07.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:07.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=278, tag_tid=23, entry_tid=32934], [object_number=277, tag_tid=23, entry_tid=32933], [object_number=276, tag_tid=23, entry_tid=32932], [object_number=279, tag_tid=23, entry_tid=32931]]], state=connected] 2026-02-22T20:03:07.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:07.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=278, tag_tid=23, entry_tid=32934' 2026-02-22T20:03:07.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=278, tag_tid=23, entry_tid=32934' 2026-02-22T20:03:07.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=278, tag_tid=23, entry_tid=32934' '!=' 'object_number=276, tag_tid=23, entry_tid=32720' 2026-02-22T20:03:07.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:07.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=278, tag_tid=23, entry_tid=32934' = 'object_number=276, tag_tid=23, entry_tid=32720' 2026-02-22T20:03:07.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=278, tag_tid=23, entry_tid=32934' 2026-02-22T20:03:07.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:07.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:07.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=278, tag_tid=23, entry_tid=32934' 2026-02-22T20:03:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:07.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:07.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:07.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:07.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=278, tag_tid=23, entry_tid=32934' 2026-02-22T20:03:07.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:07.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:07.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=32934 2026-02-22T20:03:07.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 32934 2026-02-22T20:03:07.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:07.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:07.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:08.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:08.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:08.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:08.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:08.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:08.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:08.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:08.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:08.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:08.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:08.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:08.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:08.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:08.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=33136], [object_number=279, tag_tid=23, entry_tid=33135], [object_number=278, tag_tid=23, entry_tid=33134], [object_number=277, tag_tid=23, entry_tid=33133]]], state=connected] 2026-02-22T20:03:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:08.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:08.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:08.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:08.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:08.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:08.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:08.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=33136], [object_number=279, tag_tid=23, entry_tid=33135], [object_number=278, tag_tid=23, entry_tid=33134], [object_number=277, tag_tid=23, entry_tid=33133]]], state=connected] 2026-02-22T20:03:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=276, tag_tid=23, entry_tid=33136' 2026-02-22T20:03:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=276, tag_tid=23, entry_tid=33136' 2026-02-22T20:03:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=33136' '!=' 'object_number=278, tag_tid=23, entry_tid=32934' 2026-02-22T20:03:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=33136' = 'object_number=278, tag_tid=23, entry_tid=32934' 2026-02-22T20:03:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=276, tag_tid=23, entry_tid=33136' 2026-02-22T20:03:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:08.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:08.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=33136' 2026-02-22T20:03:08.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:08.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:08.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:08.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:08.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:08.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:08.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:08.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=33136' 2026-02-22T20:03:08.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:08.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33136 2026-02-22T20:03:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 33136 2026-02-22T20:03:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:08.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:08.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:08.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:08.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:08.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:08.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:08.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:08.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:08.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:08.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:08.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:08.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:08.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:08.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:08.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:08.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:08.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:08.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:08.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:08.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:08.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:08.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:08.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:08.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=278, tag_tid=23, entry_tid=33318], [object_number=277, tag_tid=23, entry_tid=33317], [object_number=276, tag_tid=23, entry_tid=33316], [object_number=279, tag_tid=23, entry_tid=33315]]], state=connected] 2026-02-22T20:03:08.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:08.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:08.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:08.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:08.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:08.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:08.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:08.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:08.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:08.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:08.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=278, tag_tid=23, entry_tid=33318], [object_number=277, tag_tid=23, entry_tid=33317], [object_number=276, tag_tid=23, entry_tid=33316], [object_number=279, tag_tid=23, entry_tid=33315]]], state=connected] 2026-02-22T20:03:08.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=278, tag_tid=23, entry_tid=33318' 2026-02-22T20:03:08.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=278, tag_tid=23, entry_tid=33318' 2026-02-22T20:03:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=278, tag_tid=23, entry_tid=33318' '!=' 'object_number=276, tag_tid=23, entry_tid=33136' 2026-02-22T20:03:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=278, tag_tid=23, entry_tid=33318' = 'object_number=276, tag_tid=23, entry_tid=33136' 2026-02-22T20:03:08.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=278, tag_tid=23, entry_tid=33318' 2026-02-22T20:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:08.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=278, tag_tid=23, entry_tid=33318' 2026-02-22T20:03:08.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:08.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:08.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:08.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:08.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:08.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:08.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=278, tag_tid=23, entry_tid=33318' 2026-02-22T20:03:08.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:08.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:08.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33318 2026-02-22T20:03:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 33318 2026-02-22T20:03:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:08.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:08.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:08.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:08.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:08.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:08.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:08.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:08.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=33509], [object_number=276, tag_tid=23, entry_tid=33508], [object_number=279, tag_tid=23, entry_tid=33507], [object_number=278, tag_tid=23, entry_tid=33506]]], state=connected] 2026-02-22T20:03:08.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:08.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:08.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:08.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:08.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:08.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:08.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:08.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:08.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:08.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:08.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:08.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:08.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:09.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:09.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:09.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:09.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=33509], [object_number=276, tag_tid=23, entry_tid=33508], [object_number=279, tag_tid=23, entry_tid=33507], [object_number=278, tag_tid=23, entry_tid=33506]]], state=connected] 2026-02-22T20:03:09.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=277, tag_tid=23, entry_tid=33509' 2026-02-22T20:03:09.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=277, tag_tid=23, entry_tid=33509' 2026-02-22T20:03:09.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=33509' '!=' 'object_number=278, tag_tid=23, entry_tid=33318' 2026-02-22T20:03:09.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:09.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=33509' = 'object_number=278, tag_tid=23, entry_tid=33318' 2026-02-22T20:03:09.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=277, tag_tid=23, entry_tid=33509' 2026-02-22T20:03:09.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:09.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:09.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=33509' 2026-02-22T20:03:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:09.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:09.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:09.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:09.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=33509' 2026-02-22T20:03:09.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:09.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33509 2026-02-22T20:03:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 33509 2026-02-22T20:03:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:09.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:09.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:09.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:09.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:09.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:09.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:09.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:09.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:09.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:09.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:09.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:09.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:09.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:09.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:09.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:09.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:09.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:09.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:09.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:09.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=33712], [object_number=279, tag_tid=23, entry_tid=33711], [object_number=278, tag_tid=23, entry_tid=33710], [object_number=277, tag_tid=23, entry_tid=33709]]], state=connected] 2026-02-22T20:03:09.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:09.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:09.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:09.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:09.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:09.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:09.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=33712], [object_number=279, tag_tid=23, entry_tid=33711], [object_number=278, tag_tid=23, entry_tid=33710], [object_number=277, tag_tid=23, entry_tid=33709]]], state=connected] 2026-02-22T20:03:09.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=276, tag_tid=23, entry_tid=33712' 2026-02-22T20:03:09.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=276, tag_tid=23, entry_tid=33712' 2026-02-22T20:03:09.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=33712' '!=' 'object_number=277, tag_tid=23, entry_tid=33509' 2026-02-22T20:03:09.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:09.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=33712' = 'object_number=277, tag_tid=23, entry_tid=33509' 2026-02-22T20:03:09.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=276, tag_tid=23, entry_tid=33712' 2026-02-22T20:03:09.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:09.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:09.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:09.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=33712' 2026-02-22T20:03:09.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:09.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:09.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:09.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:09.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:09.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=33712' 2026-02-22T20:03:09.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:09.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33712 2026-02-22T20:03:09.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 33712 2026-02-22T20:03:09.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:09.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:09.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:09.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:09.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:09.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:09.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:09.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:09.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:09.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:09.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:09.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:09.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:09.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:09.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:09.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:09.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=278, tag_tid=23, entry_tid=33906], [object_number=277, tag_tid=23, entry_tid=33905], [object_number=276, tag_tid=23, entry_tid=33904], [object_number=279, tag_tid=23, entry_tid=33903]]], state=connected] 2026-02-22T20:03:09.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:09.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:09.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:09.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:09.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:09.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:09.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:09.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=278, tag_tid=23, entry_tid=33906], [object_number=277, tag_tid=23, entry_tid=33905], [object_number=276, tag_tid=23, entry_tid=33904], [object_number=279, tag_tid=23, entry_tid=33903]]], state=connected] 2026-02-22T20:03:09.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:09.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=278, tag_tid=23, entry_tid=33906' 2026-02-22T20:03:09.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=278, tag_tid=23, entry_tid=33906' 2026-02-22T20:03:09.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=278, tag_tid=23, entry_tid=33906' '!=' 'object_number=276, tag_tid=23, entry_tid=33712' 2026-02-22T20:03:09.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:09.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=278, tag_tid=23, entry_tid=33906' = 'object_number=276, tag_tid=23, entry_tid=33712' 2026-02-22T20:03:09.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=278, tag_tid=23, entry_tid=33906' 2026-02-22T20:03:09.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:09.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:09.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=278, tag_tid=23, entry_tid=33906' 2026-02-22T20:03:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:09.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:09.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:09.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:09.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=278, tag_tid=23, entry_tid=33906' 2026-02-22T20:03:09.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:09.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:09.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33906 2026-02-22T20:03:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 33906 2026-02-22T20:03:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:09.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:10.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:10.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:10.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:10.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:10.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:10.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=279, tag_tid=23, entry_tid=34151], [object_number=278, tag_tid=23, entry_tid=34150], [object_number=277, tag_tid=23, entry_tid=34149], [object_number=276, tag_tid=23, entry_tid=34148]]], state=connected] 2026-02-22T20:03:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:10.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:10.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:10.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:10.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:10.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=279, tag_tid=23, entry_tid=34151], [object_number=278, tag_tid=23, entry_tid=34150], [object_number=277, tag_tid=23, entry_tid=34149], [object_number=276, tag_tid=23, entry_tid=34148]]], state=connected] 2026-02-22T20:03:10.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=279, tag_tid=23, entry_tid=34151' 2026-02-22T20:03:10.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=279, tag_tid=23, entry_tid=34151' 2026-02-22T20:03:10.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=279, tag_tid=23, entry_tid=34151' '!=' 'object_number=278, tag_tid=23, entry_tid=33906' 2026-02-22T20:03:10.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:10.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=279, tag_tid=23, entry_tid=34151' = 'object_number=278, tag_tid=23, entry_tid=33906' 2026-02-22T20:03:10.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=279, tag_tid=23, entry_tid=34151' 2026-02-22T20:03:10.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:10.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:10.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:10.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=279, tag_tid=23, entry_tid=34151' 2026-02-22T20:03:10.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:10.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:10.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:10.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=279, tag_tid=23, entry_tid=34151' 2026-02-22T20:03:10.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:10.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=34151 2026-02-22T20:03:10.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 34151 2026-02-22T20:03:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:10.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:10.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:10.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:10.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:10.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:10.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:10.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:10.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:10.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:10.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:10.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:10.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:10.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:10.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:10.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:10.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:10.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:10.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:10.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:10.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:10.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:10.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:10.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:10.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:10.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:10.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:10.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:10.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:10.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:10.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:10.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=34369], [object_number=276, tag_tid=23, entry_tid=34368], [object_number=279, tag_tid=23, entry_tid=34367], [object_number=278, tag_tid=23, entry_tid=34366]]], state=connected] 2026-02-22T20:03:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:10.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:10.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:10.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:10.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:10.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:10.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:10.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:10.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:10.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:10.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=34369], [object_number=276, tag_tid=23, entry_tid=34368], [object_number=279, tag_tid=23, entry_tid=34367], [object_number=278, tag_tid=23, entry_tid=34366]]], state=connected] 2026-02-22T20:03:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=277, tag_tid=23, entry_tid=34369' 2026-02-22T20:03:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=277, tag_tid=23, entry_tid=34369' 2026-02-22T20:03:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=34369' '!=' 'object_number=279, tag_tid=23, entry_tid=34151' 2026-02-22T20:03:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=34369' = 'object_number=279, tag_tid=23, entry_tid=34151' 2026-02-22T20:03:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=277, tag_tid=23, entry_tid=34369' 2026-02-22T20:03:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=34369' 2026-02-22T20:03:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:10.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=34369' 2026-02-22T20:03:10.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:10.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=34369 2026-02-22T20:03:10.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 34369 2026-02-22T20:03:10.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:10.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:10.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:10.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:10.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:10.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:10.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:10.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:10.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:10.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:10.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=34577], [object_number=276, tag_tid=23, entry_tid=34576], [object_number=279, tag_tid=23, entry_tid=34575], [object_number=278, tag_tid=23, entry_tid=34574]]], state=connected] 2026-02-22T20:03:10.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:10.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:10.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:10.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:10.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:10.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:10.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:10.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:10.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:10.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:10.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:10.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=34577], [object_number=276, tag_tid=23, entry_tid=34576], [object_number=279, tag_tid=23, entry_tid=34575], [object_number=278, tag_tid=23, entry_tid=34574]]], state=connected] 2026-02-22T20:03:10.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=277, tag_tid=23, entry_tid=34577' 2026-02-22T20:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=277, tag_tid=23, entry_tid=34577' 2026-02-22T20:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=34577' '!=' 'object_number=277, tag_tid=23, entry_tid=34369' 2026-02-22T20:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=34577' = 'object_number=277, tag_tid=23, entry_tid=34369' 2026-02-22T20:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=277, tag_tid=23, entry_tid=34577' 2026-02-22T20:03:10.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:10.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:10.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:10.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=34577' 2026-02-22T20:03:10.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:10.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:10.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:10.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:10.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:10.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=34577' 2026-02-22T20:03:10.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:10.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:10.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=34577 2026-02-22T20:03:10.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 34577 2026-02-22T20:03:10.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:10.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:10.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:11.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:11.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:11.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:11.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:11.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:11.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:11.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:11.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:11.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:11.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:11.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:11.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:11.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:11.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:11.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:11.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:11.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:11.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:11.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:11.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:11.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:11.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:11.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:11.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:11.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:11.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:11.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=34789], [object_number=276, tag_tid=23, entry_tid=34788], [object_number=279, tag_tid=23, entry_tid=34787], [object_number=278, tag_tid=23, entry_tid=34786]]], state=connected] 2026-02-22T20:03:11.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:11.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:11.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:11.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:11.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:11.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:11.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:11.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:11.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=34789], [object_number=276, tag_tid=23, entry_tid=34788], [object_number=279, tag_tid=23, entry_tid=34787], [object_number=278, tag_tid=23, entry_tid=34786]]], state=connected] 2026-02-22T20:03:11.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=277, tag_tid=23, entry_tid=34789' 2026-02-22T20:03:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=277, tag_tid=23, entry_tid=34789' 2026-02-22T20:03:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=34789' '!=' 'object_number=277, tag_tid=23, entry_tid=34577' 2026-02-22T20:03:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=34789' = 'object_number=277, tag_tid=23, entry_tid=34577' 2026-02-22T20:03:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=277, tag_tid=23, entry_tid=34789' 2026-02-22T20:03:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:11.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:11.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=34789' 2026-02-22T20:03:11.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:11.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:11.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:11.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:11.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:11.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:11.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=34789' 2026-02-22T20:03:11.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:11.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=34789 2026-02-22T20:03:11.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 34789 2026-02-22T20:03:11.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:11.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:11.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:11.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:11.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:11.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:11.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:11.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:11.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:11.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:11.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:11.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:11.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:11.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:11.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:11.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:11.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:11.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:11.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:11.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:11.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:11.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:11.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:11.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:11.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:11.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:11.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:11.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:11.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:11.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=35028], [object_number=279, tag_tid=23, entry_tid=35027], [object_number=278, tag_tid=23, entry_tid=35026], [object_number=277, tag_tid=23, entry_tid=35025]]], state=connected] 2026-02-22T20:03:11.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:11.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:11.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:11.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:11.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:11.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:11.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:11.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:11.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=35028], [object_number=279, tag_tid=23, entry_tid=35027], [object_number=278, tag_tid=23, entry_tid=35026], [object_number=277, tag_tid=23, entry_tid=35025]]], state=connected] 2026-02-22T20:03:11.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=276, tag_tid=23, entry_tid=35028' 2026-02-22T20:03:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=276, tag_tid=23, entry_tid=35028' 2026-02-22T20:03:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=35028' '!=' 'object_number=277, tag_tid=23, entry_tid=34789' 2026-02-22T20:03:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=35028' = 'object_number=277, tag_tid=23, entry_tid=34789' 2026-02-22T20:03:11.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=276, tag_tid=23, entry_tid=35028' 2026-02-22T20:03:11.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:11.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:11.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:11.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:11.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=35028' 2026-02-22T20:03:11.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:11.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:11.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:11.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:11.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:11.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=35028' 2026-02-22T20:03:11.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:11.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:11.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35028 2026-02-22T20:03:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 35028 2026-02-22T20:03:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:11.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:11.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:11.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:11.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:11.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:11.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:11.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:11.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:11.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:11.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:11.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=35245], [object_number=276, tag_tid=23, entry_tid=35244], [object_number=279, tag_tid=23, entry_tid=35243], [object_number=278, tag_tid=23, entry_tid=35242]]], state=connected] 2026-02-22T20:03:11.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:11.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:11.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:11.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:11.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:11.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:11.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:11.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:11.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:11.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:11.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:11.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:11.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=35245], [object_number=276, tag_tid=23, entry_tid=35244], [object_number=279, tag_tid=23, entry_tid=35243], [object_number=278, tag_tid=23, entry_tid=35242]]], state=connected] 2026-02-22T20:03:12.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=277, tag_tid=23, entry_tid=35245' 2026-02-22T20:03:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=277, tag_tid=23, entry_tid=35245' 2026-02-22T20:03:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=35245' '!=' 'object_number=276, tag_tid=23, entry_tid=35028' 2026-02-22T20:03:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=35245' = 'object_number=276, tag_tid=23, entry_tid=35028' 2026-02-22T20:03:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=277, tag_tid=23, entry_tid=35245' 2026-02-22T20:03:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:12.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:12.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:12.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=35245' 2026-02-22T20:03:12.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:12.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:12.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:12.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:12.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:12.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=35245' 2026-02-22T20:03:12.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:12.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35245 2026-02-22T20:03:12.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 35245 2026-02-22T20:03:12.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:12.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:12.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:12.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:12.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:12.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:12.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:12.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:12.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:12.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:12.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:12.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:12.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:12.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:12.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:12.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:12.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:12.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:12.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:12.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:12.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:12.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:12.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:12.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:12.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:12.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:12.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:12.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:12.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:12.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:12.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:12.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:12.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:12.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:12.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:12.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:12.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:12.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:12.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:12.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=35452], [object_number=279, tag_tid=23, entry_tid=35451], [object_number=278, tag_tid=23, entry_tid=35450], [object_number=277, tag_tid=23, entry_tid=35449]]], state=connected] 2026-02-22T20:03:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:12.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:12.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:12.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:12.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:12.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=276, tag_tid=23, entry_tid=35452], [object_number=279, tag_tid=23, entry_tid=35451], [object_number=278, tag_tid=23, entry_tid=35450], [object_number=277, tag_tid=23, entry_tid=35449]]], state=connected] 2026-02-22T20:03:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=276, tag_tid=23, entry_tid=35452' 2026-02-22T20:03:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=276, tag_tid=23, entry_tid=35452' 2026-02-22T20:03:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=35452' '!=' 'object_number=277, tag_tid=23, entry_tid=35245' 2026-02-22T20:03:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=276, tag_tid=23, entry_tid=35452' = 'object_number=277, tag_tid=23, entry_tid=35245' 2026-02-22T20:03:12.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=276, tag_tid=23, entry_tid=35452' 2026-02-22T20:03:12.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:12.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:12.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=35452' 2026-02-22T20:03:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:12.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:12.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:12.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:12.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=276, tag_tid=23, entry_tid=35452' 2026-02-22T20:03:12.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:12.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35452 2026-02-22T20:03:12.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 35452 2026-02-22T20:03:12.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:12.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:12.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:12.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:12.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:12.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:12.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:12.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:12.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:12.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:12.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:12.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:12.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:12.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:12.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:12.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:12.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:12.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:12.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:12.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:12.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:12.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:12.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:12.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:12.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:12.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:12.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:12.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:12.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:12.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:12.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:12.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:12.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:12.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:12.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:12.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:12.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:12.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:12.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:12.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:12.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:12.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:12.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:12.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:12.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:12.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:12.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:12.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:12.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:12.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:12.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:12.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=279, tag_tid=23, entry_tid=35667], [object_number=278, tag_tid=23, entry_tid=35666], [object_number=277, tag_tid=23, entry_tid=35665], [object_number=276, tag_tid=23, entry_tid=35664]]], state=connected] 2026-02-22T20:03:12.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:12.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:12.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:12.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:12.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:12.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:12.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=279, tag_tid=23, entry_tid=35667], [object_number=278, tag_tid=23, entry_tid=35666], [object_number=277, tag_tid=23, entry_tid=35665], [object_number=276, tag_tid=23, entry_tid=35664]]], state=connected] 2026-02-22T20:03:12.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=279, tag_tid=23, entry_tid=35667' 2026-02-22T20:03:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=279, tag_tid=23, entry_tid=35667' 2026-02-22T20:03:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=279, tag_tid=23, entry_tid=35667' '!=' 'object_number=276, tag_tid=23, entry_tid=35452' 2026-02-22T20:03:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=279, tag_tid=23, entry_tid=35667' = 'object_number=276, tag_tid=23, entry_tid=35452' 2026-02-22T20:03:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=279, tag_tid=23, entry_tid=35667' 2026-02-22T20:03:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=279, tag_tid=23, entry_tid=35667' 2026-02-22T20:03:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:12.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:12.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:12.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=279, tag_tid=23, entry_tid=35667' 2026-02-22T20:03:12.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:12.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35667 2026-02-22T20:03:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 35667 2026-02-22T20:03:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:12.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:12.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:12.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:12.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:12.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:12.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:12.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:12.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:12.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:13.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:13.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:13.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=279, tag_tid=23, entry_tid=35883], [object_number=278, tag_tid=23, entry_tid=35882], [object_number=277, tag_tid=23, entry_tid=35881], [object_number=276, tag_tid=23, entry_tid=35880]]], state=connected] 2026-02-22T20:03:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:13.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:13.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:13.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:13.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:13.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:13.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:13.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:13.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:13.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:13.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:13.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:13.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=279, tag_tid=23, entry_tid=35883], [object_number=278, tag_tid=23, entry_tid=35882], [object_number=277, tag_tid=23, entry_tid=35881], [object_number=276, tag_tid=23, entry_tid=35880]]], state=connected] 2026-02-22T20:03:13.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=279, tag_tid=23, entry_tid=35883' 2026-02-22T20:03:13.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=279, tag_tid=23, entry_tid=35883' 2026-02-22T20:03:13.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=279, tag_tid=23, entry_tid=35883' '!=' 'object_number=279, tag_tid=23, entry_tid=35667' 2026-02-22T20:03:13.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:13.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=279, tag_tid=23, entry_tid=35883' = 'object_number=279, tag_tid=23, entry_tid=35667' 2026-02-22T20:03:13.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=279, tag_tid=23, entry_tid=35883' 2026-02-22T20:03:13.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:13.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:13.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=279, tag_tid=23, entry_tid=35883' 2026-02-22T20:03:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=279, tag_tid=23, entry_tid=35883' 2026-02-22T20:03:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35883 2026-02-22T20:03:13.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 35883 2026-02-22T20:03:13.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:13.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:13.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:13.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:13.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:13.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:13.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:13.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:13.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:13.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:13.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:13.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:13.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:13.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:13.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:13.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=36125], [object_number=276, tag_tid=23, entry_tid=36124], [object_number=279, tag_tid=23, entry_tid=36123], [object_number=278, tag_tid=23, entry_tid=36122]]], state=connected] 2026-02-22T20:03:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:13.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:13.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:13.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:13.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:13.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:13.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:13.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:13.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 69 2026-02-22T20:03:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=277, tag_tid=23, entry_tid=36125], [object_number=276, tag_tid=23, entry_tid=36124], [object_number=279, tag_tid=23, entry_tid=36123], [object_number=278, tag_tid=23, entry_tid=36122]]], state=connected] 2026-02-22T20:03:13.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=277, tag_tid=23, entry_tid=36125' 2026-02-22T20:03:13.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=277, tag_tid=23, entry_tid=36125' 2026-02-22T20:03:13.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=36125' '!=' 'object_number=279, tag_tid=23, entry_tid=35883' 2026-02-22T20:03:13.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:13.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=277, tag_tid=23, entry_tid=36125' = 'object_number=279, tag_tid=23, entry_tid=35883' 2026-02-22T20:03:13.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=277, tag_tid=23, entry_tid=36125' 2026-02-22T20:03:13.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:13.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:13.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:13.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=36125' 2026-02-22T20:03:13.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:13.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:13.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:13.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:13.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:13.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=277, tag_tid=23, entry_tid=36125' 2026-02-22T20:03:13.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:13.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36125 2026-02-22T20:03:13.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 36125 2026-02-22T20:03:13.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:13.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:13.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:13.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:13.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:13.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:13.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:13.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:13.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:13.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:13.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:13.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:13.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:13.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:13.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:13.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:13.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:13.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:13.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:13.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:13.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:13.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:13.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:13.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=36235], [object_number=282, tag_tid=23, entry_tid=36234], [object_number=281, tag_tid=23, entry_tid=36233], [object_number=280, tag_tid=23, entry_tid=36232]]], state=connected] 2026-02-22T20:03:13.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:13.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:13.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:13.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:13.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:13.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:13.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=36235], [object_number=282, tag_tid=23, entry_tid=36234], [object_number=281, tag_tid=23, entry_tid=36233], [object_number=280, tag_tid=23, entry_tid=36232]]], state=connected] 2026-02-22T20:03:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:13.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=36235' 2026-02-22T20:03:13.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=36235' 2026-02-22T20:03:13.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=36235' '!=' 'object_number=277, tag_tid=23, entry_tid=36125' 2026-02-22T20:03:13.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:13.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=36235' = 'object_number=277, tag_tid=23, entry_tid=36125' 2026-02-22T20:03:13.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=36235' 2026-02-22T20:03:13.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:13.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:13.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:13.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=36235' 2026-02-22T20:03:13.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:13.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:13.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:13.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:13.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=36235' 2026-02-22T20:03:13.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:13.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36235 2026-02-22T20:03:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 36235 2026-02-22T20:03:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:14.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:14.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:14.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:14.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:14.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:14.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:14.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:14.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:14.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:14.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:14.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:14.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:14.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=280, tag_tid=23, entry_tid=36436], [object_number=283, tag_tid=23, entry_tid=36435], [object_number=282, tag_tid=23, entry_tid=36434], [object_number=281, tag_tid=23, entry_tid=36433]]], state=connected] 2026-02-22T20:03:14.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:14.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:14.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:14.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:14.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=280, tag_tid=23, entry_tid=36436], [object_number=283, tag_tid=23, entry_tid=36435], [object_number=282, tag_tid=23, entry_tid=36434], [object_number=281, tag_tid=23, entry_tid=36433]]], state=connected] 2026-02-22T20:03:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=280, tag_tid=23, entry_tid=36436' 2026-02-22T20:03:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=280, tag_tid=23, entry_tid=36436' 2026-02-22T20:03:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=280, tag_tid=23, entry_tid=36436' '!=' 'object_number=283, tag_tid=23, entry_tid=36235' 2026-02-22T20:03:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=280, tag_tid=23, entry_tid=36436' = 'object_number=283, tag_tid=23, entry_tid=36235' 2026-02-22T20:03:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=280, tag_tid=23, entry_tid=36436' 2026-02-22T20:03:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:14.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:14.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=280, tag_tid=23, entry_tid=36436' 2026-02-22T20:03:14.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:14.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:14.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=280, tag_tid=23, entry_tid=36436' 2026-02-22T20:03:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:14.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36436 2026-02-22T20:03:14.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 36436 2026-02-22T20:03:14.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:14.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:14.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:14.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:14.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:14.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:14.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:14.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:14.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:14.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:14.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:14.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:14.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:14.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:14.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:14.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:14.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:14.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:14.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:14.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:14.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:14.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=36627], [object_number=282, tag_tid=23, entry_tid=36626], [object_number=281, tag_tid=23, entry_tid=36625], [object_number=280, tag_tid=23, entry_tid=36624]]], state=connected] 2026-02-22T20:03:14.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:14.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:14.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:14.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:14.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:14.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:14.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:14.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:14.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:14.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=36627], [object_number=282, tag_tid=23, entry_tid=36626], [object_number=281, tag_tid=23, entry_tid=36625], [object_number=280, tag_tid=23, entry_tid=36624]]], state=connected] 2026-02-22T20:03:14.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:14.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=36627' 2026-02-22T20:03:14.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=36627' 2026-02-22T20:03:14.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=36627' '!=' 'object_number=280, tag_tid=23, entry_tid=36436' 2026-02-22T20:03:14.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:14.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=36627' = 'object_number=280, tag_tid=23, entry_tid=36436' 2026-02-22T20:03:14.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=36627' 2026-02-22T20:03:14.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:14.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:14.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:14.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:14.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=36627' 2026-02-22T20:03:14.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:14.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:14.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:14.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:14.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:14.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=36627' 2026-02-22T20:03:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:14.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36627 2026-02-22T20:03:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 36627 2026-02-22T20:03:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:14.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:15.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:15.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=282, tag_tid=23, entry_tid=36950], [object_number=281, tag_tid=23, entry_tid=36949], [object_number=280, tag_tid=23, entry_tid=36948], [object_number=283, tag_tid=23, entry_tid=36947]]], state=connected] 2026-02-22T20:03:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:15.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:15.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:15.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:15.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=282, tag_tid=23, entry_tid=36950], [object_number=281, tag_tid=23, entry_tid=36949], [object_number=280, tag_tid=23, entry_tid=36948], [object_number=283, tag_tid=23, entry_tid=36947]]], state=connected] 2026-02-22T20:03:15.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=282, tag_tid=23, entry_tid=36950' 2026-02-22T20:03:15.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=282, tag_tid=23, entry_tid=36950' 2026-02-22T20:03:15.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=282, tag_tid=23, entry_tid=36950' '!=' 'object_number=283, tag_tid=23, entry_tid=36627' 2026-02-22T20:03:15.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:15.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=282, tag_tid=23, entry_tid=36950' = 'object_number=283, tag_tid=23, entry_tid=36627' 2026-02-22T20:03:15.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=282, tag_tid=23, entry_tid=36950' 2026-02-22T20:03:15.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:15.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:15.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:15.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=282, tag_tid=23, entry_tid=36950' 2026-02-22T20:03:15.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:15.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=282, tag_tid=23, entry_tid=36950' 2026-02-22T20:03:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36950 2026-02-22T20:03:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 36950 2026-02-22T20:03:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:15.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:15.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:15.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:15.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:15.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:15.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:15.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:15.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:15.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:15.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:15.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:15.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:15.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:15.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:15.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:15.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:15.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=37155], [object_number=282, tag_tid=23, entry_tid=37154], [object_number=281, tag_tid=23, entry_tid=37153], [object_number=280, tag_tid=23, entry_tid=37152]]], state=connected] 2026-02-22T20:03:15.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:15.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:15.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:15.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:15.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:15.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:15.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=37155], [object_number=282, tag_tid=23, entry_tid=37154], [object_number=281, tag_tid=23, entry_tid=37153], [object_number=280, tag_tid=23, entry_tid=37152]]], state=connected] 2026-02-22T20:03:15.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=37155' 2026-02-22T20:03:15.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=37155' 2026-02-22T20:03:15.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=37155' '!=' 'object_number=282, tag_tid=23, entry_tid=36950' 2026-02-22T20:03:15.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:15.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=37155' = 'object_number=282, tag_tid=23, entry_tid=36950' 2026-02-22T20:03:15.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=37155' 2026-02-22T20:03:15.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:15.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:15.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:15.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=37155' 2026-02-22T20:03:15.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:15.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:15.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:15.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:15.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:15.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=37155' 2026-02-22T20:03:15.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:15.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37155 2026-02-22T20:03:15.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 37155 2026-02-22T20:03:15.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:15.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:15.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:15.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:15.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:15.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:15.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:15.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:15.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:15.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:15.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:15.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:15.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:15.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:15.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:15.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:15.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:15.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:15.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:15.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:15.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:15.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:15.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:15.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:15.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:15.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:15.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:15.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:15.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:15.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:15.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=37377], [object_number=280, tag_tid=23, entry_tid=37376], [object_number=283, tag_tid=23, entry_tid=37375], [object_number=282, tag_tid=23, entry_tid=37374]]], state=connected] 2026-02-22T20:03:15.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:15.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:15.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:15.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:15.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:15.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:15.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:15.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:15.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:15.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:15.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:15.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=37377], [object_number=280, tag_tid=23, entry_tid=37376], [object_number=283, tag_tid=23, entry_tid=37375], [object_number=282, tag_tid=23, entry_tid=37374]]], state=connected] 2026-02-22T20:03:15.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:15.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=281, tag_tid=23, entry_tid=37377' 2026-02-22T20:03:15.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=281, tag_tid=23, entry_tid=37377' 2026-02-22T20:03:15.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=37377' '!=' 'object_number=283, tag_tid=23, entry_tid=37155' 2026-02-22T20:03:15.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:15.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=37377' = 'object_number=283, tag_tid=23, entry_tid=37155' 2026-02-22T20:03:15.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=281, tag_tid=23, entry_tid=37377' 2026-02-22T20:03:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:15.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=37377' 2026-02-22T20:03:15.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:15.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:15.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:15.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:15.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:15.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=37377' 2026-02-22T20:03:15.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:15.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:15.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37377 2026-02-22T20:03:15.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 37377 2026-02-22T20:03:15.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:15.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:15.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:16.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:16.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:16.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:16.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:16.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:16.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:16.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:16.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:16.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:16.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=37583], [object_number=282, tag_tid=23, entry_tid=37582], [object_number=281, tag_tid=23, entry_tid=37581], [object_number=280, tag_tid=23, entry_tid=37580]]], state=connected] 2026-02-22T20:03:16.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:16.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:16.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:16.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:16.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:16.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:16.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=37583], [object_number=282, tag_tid=23, entry_tid=37582], [object_number=281, tag_tid=23, entry_tid=37581], [object_number=280, tag_tid=23, entry_tid=37580]]], state=connected] 2026-02-22T20:03:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=37583' 2026-02-22T20:03:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=37583' 2026-02-22T20:03:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=37583' '!=' 'object_number=281, tag_tid=23, entry_tid=37377' 2026-02-22T20:03:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=37583' = 'object_number=281, tag_tid=23, entry_tid=37377' 2026-02-22T20:03:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=37583' 2026-02-22T20:03:16.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:16.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:16.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:16.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=37583' 2026-02-22T20:03:16.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:16.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:16.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:16.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:16.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:16.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=37583' 2026-02-22T20:03:16.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:16.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37583 2026-02-22T20:03:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 37583 2026-02-22T20:03:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:16.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:16.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:16.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:16.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:16.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:16.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:16.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:16.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:16.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:16.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:16.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:16.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:16.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:16.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:16.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:16.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:16.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:16.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:16.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:16.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:16.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:16.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:16.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:16.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:16.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:16.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:16.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:16.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=282, tag_tid=23, entry_tid=37786], [object_number=281, tag_tid=23, entry_tid=37785], [object_number=280, tag_tid=23, entry_tid=37784], [object_number=283, tag_tid=23, entry_tid=37783]]], state=connected] 2026-02-22T20:03:16.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:16.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:16.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:16.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:16.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:16.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:16.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:16.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:16.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=282, tag_tid=23, entry_tid=37786], [object_number=281, tag_tid=23, entry_tid=37785], [object_number=280, tag_tid=23, entry_tid=37784], [object_number=283, tag_tid=23, entry_tid=37783]]], state=connected] 2026-02-22T20:03:16.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=282, tag_tid=23, entry_tid=37786' 2026-02-22T20:03:16.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=282, tag_tid=23, entry_tid=37786' 2026-02-22T20:03:16.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=282, tag_tid=23, entry_tid=37786' '!=' 'object_number=283, tag_tid=23, entry_tid=37583' 2026-02-22T20:03:16.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:16.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=282, tag_tid=23, entry_tid=37786' = 'object_number=283, tag_tid=23, entry_tid=37583' 2026-02-22T20:03:16.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=282, tag_tid=23, entry_tid=37786' 2026-02-22T20:03:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:16.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=282, tag_tid=23, entry_tid=37786' 2026-02-22T20:03:16.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:16.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:16.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:16.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:16.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:16.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=282, tag_tid=23, entry_tid=37786' 2026-02-22T20:03:16.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:16.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37786 2026-02-22T20:03:16.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 37786 2026-02-22T20:03:16.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:16.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:16.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:16.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:16.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:16.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:16.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:16.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:16.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:16.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:16.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:16.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:16.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:16.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:16.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:16.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:16.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:16.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:16.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:16.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:16.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:16.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:16.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:16.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:16.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:16.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:16.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:16.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=37983], [object_number=282, tag_tid=23, entry_tid=37982], [object_number=281, tag_tid=23, entry_tid=37981], [object_number=280, tag_tid=23, entry_tid=37980]]], state=connected] 2026-02-22T20:03:16.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:16.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:16.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:16.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:16.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:16.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:16.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:16.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=37983], [object_number=282, tag_tid=23, entry_tid=37982], [object_number=281, tag_tid=23, entry_tid=37981], [object_number=280, tag_tid=23, entry_tid=37980]]], state=connected] 2026-02-22T20:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=37983' 2026-02-22T20:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=37983' 2026-02-22T20:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=37983' '!=' 'object_number=282, tag_tid=23, entry_tid=37786' 2026-02-22T20:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=37983' = 'object_number=282, tag_tid=23, entry_tid=37786' 2026-02-22T20:03:16.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=37983' 2026-02-22T20:03:16.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:16.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:16.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:16.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=37983' 2026-02-22T20:03:16.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:16.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:16.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:16.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=37983' 2026-02-22T20:03:16.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:16.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=37983 2026-02-22T20:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 37983 2026-02-22T20:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:17.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:17.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:17.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:17.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:17.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:17.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:17.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:17.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:17.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:17.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:17.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:17.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:17.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:17.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:17.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:17.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:17.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:17.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:17.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:17.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:17.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:17.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:17.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:17.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:17.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:17.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:17.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:17.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:17.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:17.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:17.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:17.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:17.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:17.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:17.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:17.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:17.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:17.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:17.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:17.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:17.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:17.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:17.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:17.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:17.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:17.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:17.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=38181], [object_number=280, tag_tid=23, entry_tid=38180], [object_number=283, tag_tid=23, entry_tid=38179], [object_number=282, tag_tid=23, entry_tid=38178]]], state=connected] 2026-02-22T20:03:17.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:17.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:17.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:17.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:17.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:17.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:17.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:17.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:17.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:17.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:17.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:17.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:17.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:17.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:17.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:17.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:17.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:17.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:17.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=38181], [object_number=280, tag_tid=23, entry_tid=38180], [object_number=283, tag_tid=23, entry_tid=38179], [object_number=282, tag_tid=23, entry_tid=38178]]], state=connected] 2026-02-22T20:03:17.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=281, tag_tid=23, entry_tid=38181' 2026-02-22T20:03:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=281, tag_tid=23, entry_tid=38181' 2026-02-22T20:03:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=38181' '!=' 'object_number=283, tag_tid=23, entry_tid=37983' 2026-02-22T20:03:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=38181' = 'object_number=283, tag_tid=23, entry_tid=37983' 2026-02-22T20:03:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=281, tag_tid=23, entry_tid=38181' 2026-02-22T20:03:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:17.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:17.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:17.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=38181' 2026-02-22T20:03:17.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:17.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:17.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:17.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:17.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:17.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=38181' 2026-02-22T20:03:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:17.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38181 2026-02-22T20:03:17.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 38181 2026-02-22T20:03:17.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:17.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:17.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:17.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:17.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:17.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:17.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:17.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:17.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:17.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:17.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:17.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:17.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:17.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:17.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:17.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:17.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=38375], [object_number=282, tag_tid=23, entry_tid=38374], [object_number=281, tag_tid=23, entry_tid=38373], [object_number=280, tag_tid=23, entry_tid=38372]]], state=connected] 2026-02-22T20:03:17.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:17.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:17.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:17.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:17.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:17.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:17.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:17.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:17.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=38375], [object_number=282, tag_tid=23, entry_tid=38374], [object_number=281, tag_tid=23, entry_tid=38373], [object_number=280, tag_tid=23, entry_tid=38372]]], state=connected] 2026-02-22T20:03:17.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:17.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=38375' 2026-02-22T20:03:17.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=38375' 2026-02-22T20:03:17.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=38375' '!=' 'object_number=281, tag_tid=23, entry_tid=38181' 2026-02-22T20:03:17.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:17.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=38375' = 'object_number=281, tag_tid=23, entry_tid=38181' 2026-02-22T20:03:17.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=38375' 2026-02-22T20:03:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=38375' 2026-02-22T20:03:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:17.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:17.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:17.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:17.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:17.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=38375' 2026-02-22T20:03:17.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:17.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38375 2026-02-22T20:03:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 38375 2026-02-22T20:03:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:17.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:17.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:17.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:18.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:18.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:18.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:18.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:18.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:18.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:18.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:18.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:18.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:18.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:18.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:18.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:18.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=282, tag_tid=23, entry_tid=38586], [object_number=281, tag_tid=23, entry_tid=38585], [object_number=280, tag_tid=23, entry_tid=38584], [object_number=283, tag_tid=23, entry_tid=38583]]], state=connected] 2026-02-22T20:03:18.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:18.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:18.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:18.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:18.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:18.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:18.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:18.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:18.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:18.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=282, tag_tid=23, entry_tid=38586], [object_number=281, tag_tid=23, entry_tid=38585], [object_number=280, tag_tid=23, entry_tid=38584], [object_number=283, tag_tid=23, entry_tid=38583]]], state=connected] 2026-02-22T20:03:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=282, tag_tid=23, entry_tid=38586' 2026-02-22T20:03:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=282, tag_tid=23, entry_tid=38586' 2026-02-22T20:03:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=282, tag_tid=23, entry_tid=38586' '!=' 'object_number=283, tag_tid=23, entry_tid=38375' 2026-02-22T20:03:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=282, tag_tid=23, entry_tid=38586' = 'object_number=283, tag_tid=23, entry_tid=38375' 2026-02-22T20:03:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=282, tag_tid=23, entry_tid=38586' 2026-02-22T20:03:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:18.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=282, tag_tid=23, entry_tid=38586' 2026-02-22T20:03:18.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:18.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:18.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:18.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:18.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:18.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=282, tag_tid=23, entry_tid=38586' 2026-02-22T20:03:18.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:18.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38586 2026-02-22T20:03:18.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 38586 2026-02-22T20:03:18.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:18.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:18.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:18.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:18.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:18.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:18.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:18.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:18.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:18.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:18.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:18.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:18.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:18.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:18.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:18.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:18.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=38787], [object_number=282, tag_tid=23, entry_tid=38786], [object_number=281, tag_tid=23, entry_tid=38785], [object_number=280, tag_tid=23, entry_tid=38784]]], state=connected] 2026-02-22T20:03:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:18.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:18.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:18.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:18.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:18.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:18.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:18.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:18.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:18.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:18.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:18.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:18.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=38787], [object_number=282, tag_tid=23, entry_tid=38786], [object_number=281, tag_tid=23, entry_tid=38785], [object_number=280, tag_tid=23, entry_tid=38784]]], state=connected] 2026-02-22T20:03:18.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=38787' 2026-02-22T20:03:18.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=38787' 2026-02-22T20:03:18.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=38787' '!=' 'object_number=282, tag_tid=23, entry_tid=38586' 2026-02-22T20:03:18.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:18.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=38787' = 'object_number=282, tag_tid=23, entry_tid=38586' 2026-02-22T20:03:18.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=38787' 2026-02-22T20:03:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:18.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=38787' 2026-02-22T20:03:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:18.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:18.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:18.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=38787' 2026-02-22T20:03:18.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:18.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38787 2026-02-22T20:03:18.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 38787 2026-02-22T20:03:18.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:18.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:18.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:18.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:18.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:18.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:18.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:18.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:18.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:18.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:18.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:18.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:18.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:18.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:18.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:18.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:18.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:18.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:18.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:18.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:18.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:18.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:18.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:18.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:18.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:18.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:18.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:18.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:18.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=38979], [object_number=282, tag_tid=23, entry_tid=38978], [object_number=281, tag_tid=23, entry_tid=38977], [object_number=280, tag_tid=23, entry_tid=38976]]], state=connected] 2026-02-22T20:03:18.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:18.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:18.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:18.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:18.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:18.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:18.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:18.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:18.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=38979], [object_number=282, tag_tid=23, entry_tid=38978], [object_number=281, tag_tid=23, entry_tid=38977], [object_number=280, tag_tid=23, entry_tid=38976]]], state=connected] 2026-02-22T20:03:18.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:18.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=38979' 2026-02-22T20:03:18.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=38979' 2026-02-22T20:03:18.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=38979' '!=' 'object_number=283, tag_tid=23, entry_tid=38787' 2026-02-22T20:03:18.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:18.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=38979' = 'object_number=283, tag_tid=23, entry_tid=38787' 2026-02-22T20:03:18.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=38979' 2026-02-22T20:03:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:18.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=38979' 2026-02-22T20:03:18.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:18.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:18.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:18.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:18.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:18.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=38979' 2026-02-22T20:03:18.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:18.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:18.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=38979 2026-02-22T20:03:18.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 38979 2026-02-22T20:03:18.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:18.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:18.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:19.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:19.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:19.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:19.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:19.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:19.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:19.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:19.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:19.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:19.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:19.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:19.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:19.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:19.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:19.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:19.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=39163], [object_number=282, tag_tid=23, entry_tid=39162], [object_number=281, tag_tid=23, entry_tid=39161], [object_number=280, tag_tid=23, entry_tid=39160]]], state=connected] 2026-02-22T20:03:19.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:19.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:19.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:19.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:19.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:19.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:19.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:19.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:19.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=39163], [object_number=282, tag_tid=23, entry_tid=39162], [object_number=281, tag_tid=23, entry_tid=39161], [object_number=280, tag_tid=23, entry_tid=39160]]], state=connected] 2026-02-22T20:03:19.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=39163' 2026-02-22T20:03:19.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=39163' 2026-02-22T20:03:19.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=39163' '!=' 'object_number=283, tag_tid=23, entry_tid=38979' 2026-02-22T20:03:19.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:19.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=39163' = 'object_number=283, tag_tid=23, entry_tid=38979' 2026-02-22T20:03:19.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=39163' 2026-02-22T20:03:19.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:19.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:19.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:19.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=39163' 2026-02-22T20:03:19.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:19.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:19.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:19.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:19.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=39163' 2026-02-22T20:03:19.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:19.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39163 2026-02-22T20:03:19.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 39163 2026-02-22T20:03:19.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:19.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:19.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:19.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:19.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:19.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:19.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:19.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:19.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:19.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:19.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:19.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:19.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:19.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:19.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:19.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:19.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:19.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:19.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=39365], [object_number=280, tag_tid=23, entry_tid=39364], [object_number=283, tag_tid=23, entry_tid=39363], [object_number=282, tag_tid=23, entry_tid=39362]]], state=connected] 2026-02-22T20:03:19.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:19.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:19.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:19.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:19.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:19.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:19.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:19.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:19.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:19.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:19.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:19.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:19.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=39365], [object_number=280, tag_tid=23, entry_tid=39364], [object_number=283, tag_tid=23, entry_tid=39363], [object_number=282, tag_tid=23, entry_tid=39362]]], state=connected] 2026-02-22T20:03:19.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=281, tag_tid=23, entry_tid=39365' 2026-02-22T20:03:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=281, tag_tid=23, entry_tid=39365' 2026-02-22T20:03:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=39365' '!=' 'object_number=283, tag_tid=23, entry_tid=39163' 2026-02-22T20:03:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=39365' = 'object_number=283, tag_tid=23, entry_tid=39163' 2026-02-22T20:03:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=281, tag_tid=23, entry_tid=39365' 2026-02-22T20:03:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:19.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=39365' 2026-02-22T20:03:19.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:19.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:19.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:19.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:19.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:19.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=39365' 2026-02-22T20:03:19.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:19.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39365 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 39365 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:19.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:19.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:19.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:19.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:19.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:19.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:19.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:19.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:19.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:19.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:19.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:19.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:19.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:19.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:19.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:19.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:19.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:19.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:19.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:19.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:19.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=280, tag_tid=23, entry_tid=39560], [object_number=283, tag_tid=23, entry_tid=39559], [object_number=282, tag_tid=23, entry_tid=39558], [object_number=281, tag_tid=23, entry_tid=39557]]], state=connected] 2026-02-22T20:03:19.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:19.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:19.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:19.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:19.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:19.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:19.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:19.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:19.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:19.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=280, tag_tid=23, entry_tid=39560], [object_number=283, tag_tid=23, entry_tid=39559], [object_number=282, tag_tid=23, entry_tid=39558], [object_number=281, tag_tid=23, entry_tid=39557]]], state=connected] 2026-02-22T20:03:19.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=280, tag_tid=23, entry_tid=39560' 2026-02-22T20:03:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=280, tag_tid=23, entry_tid=39560' 2026-02-22T20:03:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=280, tag_tid=23, entry_tid=39560' '!=' 'object_number=281, tag_tid=23, entry_tid=39365' 2026-02-22T20:03:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=280, tag_tid=23, entry_tid=39560' = 'object_number=281, tag_tid=23, entry_tid=39365' 2026-02-22T20:03:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=280, tag_tid=23, entry_tid=39560' 2026-02-22T20:03:19.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:19.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:19.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=280, tag_tid=23, entry_tid=39560' 2026-02-22T20:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=280, tag_tid=23, entry_tid=39560' 2026-02-22T20:03:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:19.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39560 2026-02-22T20:03:19.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 39560 2026-02-22T20:03:19.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:19.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:19.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:20.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:20.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:20.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:20.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:20.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:20.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:20.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:20.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:20.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:20.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:20.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:20.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:20.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:20.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:20.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:20.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:20.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:20.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:20.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:20.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:20.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:20.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=39747], [object_number=282, tag_tid=23, entry_tid=39746], [object_number=281, tag_tid=23, entry_tid=39745], [object_number=280, tag_tid=23, entry_tid=39744]]], state=connected] 2026-02-22T20:03:20.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:20.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:20.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:20.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:20.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:20.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:20.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:20.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:20.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:20.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:20.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:20.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:20.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:20.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:20.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:20.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:20.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:20.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=39747], [object_number=282, tag_tid=23, entry_tid=39746], [object_number=281, tag_tid=23, entry_tid=39745], [object_number=280, tag_tid=23, entry_tid=39744]]], state=connected] 2026-02-22T20:03:20.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=39747' 2026-02-22T20:03:20.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=39747' 2026-02-22T20:03:20.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=39747' '!=' 'object_number=280, tag_tid=23, entry_tid=39560' 2026-02-22T20:03:20.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:20.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=39747' = 'object_number=280, tag_tid=23, entry_tid=39560' 2026-02-22T20:03:20.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=39747' 2026-02-22T20:03:20.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:20.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:20.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:20.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:20.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=39747' 2026-02-22T20:03:20.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:20.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:20.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:20.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:20.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:20.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:20.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:20.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=39747' 2026-02-22T20:03:20.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:20.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39747 2026-02-22T20:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 39747 2026-02-22T20:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:20.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:20.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:20.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:20.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:20.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:20.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:20.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:20.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:20.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:20.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:20.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:20.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:20.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:20.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:20.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:20.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:20.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:20.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=39941], [object_number=280, tag_tid=23, entry_tid=39940], [object_number=283, tag_tid=23, entry_tid=39939], [object_number=282, tag_tid=23, entry_tid=39938]]], state=connected] 2026-02-22T20:03:20.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:20.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:20.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:20.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:20.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=39941], [object_number=280, tag_tid=23, entry_tid=39940], [object_number=283, tag_tid=23, entry_tid=39939], [object_number=282, tag_tid=23, entry_tid=39938]]], state=connected] 2026-02-22T20:03:20.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=281, tag_tid=23, entry_tid=39941' 2026-02-22T20:03:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=281, tag_tid=23, entry_tid=39941' 2026-02-22T20:03:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=39941' '!=' 'object_number=283, tag_tid=23, entry_tid=39747' 2026-02-22T20:03:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=39941' = 'object_number=283, tag_tid=23, entry_tid=39747' 2026-02-22T20:03:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=281, tag_tid=23, entry_tid=39941' 2026-02-22T20:03:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:20.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:20.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:20.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=39941' 2026-02-22T20:03:20.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:20.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:20.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:20.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:20.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:20.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:20.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:20.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=39941' 2026-02-22T20:03:20.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:20.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39941 2026-02-22T20:03:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 39941 2026-02-22T20:03:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:21.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:21.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:21.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:21.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:21.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:21.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:21.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:21.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:21.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:21.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:21.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:21.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:21.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:21.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=40137], [object_number=280, tag_tid=23, entry_tid=40136], [object_number=283, tag_tid=23, entry_tid=40135], [object_number=282, tag_tid=23, entry_tid=40134]]], state=connected] 2026-02-22T20:03:21.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:21.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:21.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:21.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=281, tag_tid=23, entry_tid=40137], [object_number=280, tag_tid=23, entry_tid=40136], [object_number=283, tag_tid=23, entry_tid=40135], [object_number=282, tag_tid=23, entry_tid=40134]]], state=connected] 2026-02-22T20:03:21.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=281, tag_tid=23, entry_tid=40137' 2026-02-22T20:03:21.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=281, tag_tid=23, entry_tid=40137' 2026-02-22T20:03:21.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=40137' '!=' 'object_number=281, tag_tid=23, entry_tid=39941' 2026-02-22T20:03:21.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:21.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=281, tag_tid=23, entry_tid=40137' = 'object_number=281, tag_tid=23, entry_tid=39941' 2026-02-22T20:03:21.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=281, tag_tid=23, entry_tid=40137' 2026-02-22T20:03:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:21.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=40137' 2026-02-22T20:03:21.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:21.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:21.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:21.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:21.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:21.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=281, tag_tid=23, entry_tid=40137' 2026-02-22T20:03:21.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:21.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40137 2026-02-22T20:03:21.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 40137 2026-02-22T20:03:21.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:21.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:21.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:21.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:21.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:21.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:21.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:21.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:21.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:21.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:21.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=40347], [object_number=282, tag_tid=23, entry_tid=40346], [object_number=281, tag_tid=23, entry_tid=40345], [object_number=280, tag_tid=23, entry_tid=40344]]], state=connected] 2026-02-22T20:03:21.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:21.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:21.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:21.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:21.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 70 2026-02-22T20:03:21.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:21.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:21.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:21.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=283, tag_tid=23, entry_tid=40347], [object_number=282, tag_tid=23, entry_tid=40346], [object_number=281, tag_tid=23, entry_tid=40345], [object_number=280, tag_tid=23, entry_tid=40344]]], state=connected] 2026-02-22T20:03:21.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=283, tag_tid=23, entry_tid=40347' 2026-02-22T20:03:21.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=283, tag_tid=23, entry_tid=40347' 2026-02-22T20:03:21.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=40347' '!=' 'object_number=281, tag_tid=23, entry_tid=40137' 2026-02-22T20:03:21.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:21.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=283, tag_tid=23, entry_tid=40347' = 'object_number=281, tag_tid=23, entry_tid=40137' 2026-02-22T20:03:21.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=283, tag_tid=23, entry_tid=40347' 2026-02-22T20:03:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:21.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=40347' 2026-02-22T20:03:21.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:21.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:21.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:21.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=283, tag_tid=23, entry_tid=40347' 2026-02-22T20:03:21.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:21.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40347 2026-02-22T20:03:21.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 40347 2026-02-22T20:03:21.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:21.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:21.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:21.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:21.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:21.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:21.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=40472], [object_number=287, tag_tid=23, entry_tid=40471], [object_number=286, tag_tid=23, entry_tid=40470], [object_number=285, tag_tid=23, entry_tid=40469]]], state=connected] 2026-02-22T20:03:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:21.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:21.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:21.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:21.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:21.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:21.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:21.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:21.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:21.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:21.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:21.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:21.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:21.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=40472], [object_number=287, tag_tid=23, entry_tid=40471], [object_number=286, tag_tid=23, entry_tid=40470], [object_number=285, tag_tid=23, entry_tid=40469]]], state=connected] 2026-02-22T20:03:21.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:21.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=284, tag_tid=23, entry_tid=40472' 2026-02-22T20:03:21.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=284, tag_tid=23, entry_tid=40472' 2026-02-22T20:03:21.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=40472' '!=' 'object_number=283, tag_tid=23, entry_tid=40347' 2026-02-22T20:03:21.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:21.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=40472' = 'object_number=283, tag_tid=23, entry_tid=40347' 2026-02-22T20:03:21.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=284, tag_tid=23, entry_tid=40472' 2026-02-22T20:03:21.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:21.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:21.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:21.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=40472' 2026-02-22T20:03:21.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:21.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:21.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:21.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:21.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:21.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=40472' 2026-02-22T20:03:21.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:21.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:21.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40472 2026-02-22T20:03:21.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 40472 2026-02-22T20:03:21.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:21.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:21.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:22.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:22.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:22.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:22.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:22.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:22.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:22.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:22.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:22.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:22.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:22.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:22.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:22.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:22.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:22.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:22.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:22.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:22.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:22.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:22.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:22.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:22.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:22.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:22.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:22.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:22.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:22.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:22.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=285, tag_tid=23, entry_tid=40693], [object_number=284, tag_tid=23, entry_tid=40692], [object_number=287, tag_tid=23, entry_tid=40691], [object_number=286, tag_tid=23, entry_tid=40690]]], state=connected] 2026-02-22T20:03:22.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:22.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:22.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:22.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:22.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:22.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:22.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:22.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=285, tag_tid=23, entry_tid=40693], [object_number=284, tag_tid=23, entry_tid=40692], [object_number=287, tag_tid=23, entry_tid=40691], [object_number=286, tag_tid=23, entry_tid=40690]]], state=connected] 2026-02-22T20:03:22.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=285, tag_tid=23, entry_tid=40693' 2026-02-22T20:03:22.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=285, tag_tid=23, entry_tid=40693' 2026-02-22T20:03:22.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=285, tag_tid=23, entry_tid=40693' '!=' 'object_number=284, tag_tid=23, entry_tid=40472' 2026-02-22T20:03:22.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:22.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=285, tag_tid=23, entry_tid=40693' = 'object_number=284, tag_tid=23, entry_tid=40472' 2026-02-22T20:03:22.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=285, tag_tid=23, entry_tid=40693' 2026-02-22T20:03:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:22.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:22.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=285, tag_tid=23, entry_tid=40693' 2026-02-22T20:03:22.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:22.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:22.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:22.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:22.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:22.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:22.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=285, tag_tid=23, entry_tid=40693' 2026-02-22T20:03:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:22.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40693 2026-02-22T20:03:22.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 40693 2026-02-22T20:03:22.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:22.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:22.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:22.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:22.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:22.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:22.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:22.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:22.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:22.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:22.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:22.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:22.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:22.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:22.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:22.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:22.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:22.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:22.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:22.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:22.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:22.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:22.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:22.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:22.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:22.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:22.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:22.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:22.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:22.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:22.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:22.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:22.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:22.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:22.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:22.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:22.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:22.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:22.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:22.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:22.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:22.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:22.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=40910], [object_number=285, tag_tid=23, entry_tid=40909], [object_number=284, tag_tid=23, entry_tid=40908], [object_number=287, tag_tid=23, entry_tid=40907]]], state=connected] 2026-02-22T20:03:22.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:22.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:22.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:22.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:22.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:22.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=40910], [object_number=285, tag_tid=23, entry_tid=40909], [object_number=284, tag_tid=23, entry_tid=40908], [object_number=287, tag_tid=23, entry_tid=40907]]], state=connected] 2026-02-22T20:03:22.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:22.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=286, tag_tid=23, entry_tid=40910' 2026-02-22T20:03:22.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=286, tag_tid=23, entry_tid=40910' 2026-02-22T20:03:22.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=40910' '!=' 'object_number=285, tag_tid=23, entry_tid=40693' 2026-02-22T20:03:22.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:22.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=40910' = 'object_number=285, tag_tid=23, entry_tid=40693' 2026-02-22T20:03:22.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=286, tag_tid=23, entry_tid=40910' 2026-02-22T20:03:22.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:22.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:22.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=40910' 2026-02-22T20:03:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:22.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:22.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:22.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:22.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=40910' 2026-02-22T20:03:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40910 2026-02-22T20:03:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 40910 2026-02-22T20:03:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:22.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:22.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:22.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:22.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:22.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:22.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:22.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:22.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:22.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:22.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:22.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:22.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:22.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:22.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:22.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:22.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:23.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:23.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:23.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:23.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:23.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:23.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:23.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=285, tag_tid=23, entry_tid=41121], [object_number=284, tag_tid=23, entry_tid=41120], [object_number=287, tag_tid=23, entry_tid=41119], [object_number=286, tag_tid=23, entry_tid=41118]]], state=connected] 2026-02-22T20:03:23.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:23.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:23.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:23.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:23.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:23.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:23.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:23.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:23.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=285, tag_tid=23, entry_tid=41121], [object_number=284, tag_tid=23, entry_tid=41120], [object_number=287, tag_tid=23, entry_tid=41119], [object_number=286, tag_tid=23, entry_tid=41118]]], state=connected] 2026-02-22T20:03:23.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=285, tag_tid=23, entry_tid=41121' 2026-02-22T20:03:23.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=285, tag_tid=23, entry_tid=41121' 2026-02-22T20:03:23.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=285, tag_tid=23, entry_tid=41121' '!=' 'object_number=286, tag_tid=23, entry_tid=40910' 2026-02-22T20:03:23.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:23.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=285, tag_tid=23, entry_tid=41121' = 'object_number=286, tag_tid=23, entry_tid=40910' 2026-02-22T20:03:23.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=285, tag_tid=23, entry_tid=41121' 2026-02-22T20:03:23.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:23.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:23.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:23.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=285, tag_tid=23, entry_tid=41121' 2026-02-22T20:03:23.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:23.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:23.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=285, tag_tid=23, entry_tid=41121' 2026-02-22T20:03:23.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:23.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41121 2026-02-22T20:03:23.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 41121 2026-02-22T20:03:23.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:23.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:23.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:23.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:23.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:23.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:23.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:23.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:23.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:23.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:23.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:23.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:23.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:23.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:23.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:23.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:23.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:23.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:23.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:23.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:23.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:23.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:23.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:23.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=41323], [object_number=286, tag_tid=23, entry_tid=41322], [object_number=285, tag_tid=23, entry_tid=41321], [object_number=284, tag_tid=23, entry_tid=41320]]], state=connected] 2026-02-22T20:03:23.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:23.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:23.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:23.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:23.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:23.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:23.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:23.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:23.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:23.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:23.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=41323], [object_number=286, tag_tid=23, entry_tid=41322], [object_number=285, tag_tid=23, entry_tid=41321], [object_number=284, tag_tid=23, entry_tid=41320]]], state=connected] 2026-02-22T20:03:23.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=287, tag_tid=23, entry_tid=41323' 2026-02-22T20:03:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=287, tag_tid=23, entry_tid=41323' 2026-02-22T20:03:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=41323' '!=' 'object_number=285, tag_tid=23, entry_tid=41121' 2026-02-22T20:03:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=41323' = 'object_number=285, tag_tid=23, entry_tid=41121' 2026-02-22T20:03:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=287, tag_tid=23, entry_tid=41323' 2026-02-22T20:03:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:23.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=41323' 2026-02-22T20:03:23.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:23.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:23.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:23.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:23.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:23.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=41323' 2026-02-22T20:03:23.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:23.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41323 2026-02-22T20:03:23.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 41323 2026-02-22T20:03:23.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:23.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:23.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:23.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:23.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:23.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:23.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:23.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:23.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:23.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:23.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:23.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:23.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:23.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:23.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=41556], [object_number=287, tag_tid=23, entry_tid=41555], [object_number=286, tag_tid=23, entry_tid=41554], [object_number=285, tag_tid=23, entry_tid=41553]]], state=connected] 2026-02-22T20:03:23.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:23.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:23.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:23.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:23.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:23.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:23.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:23.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:23.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:23.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:23.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:23.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=41556], [object_number=287, tag_tid=23, entry_tid=41555], [object_number=286, tag_tid=23, entry_tid=41554], [object_number=285, tag_tid=23, entry_tid=41553]]], state=connected] 2026-02-22T20:03:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:23.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=284, tag_tid=23, entry_tid=41556' 2026-02-22T20:03:23.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=284, tag_tid=23, entry_tid=41556' 2026-02-22T20:03:23.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=41556' '!=' 'object_number=287, tag_tid=23, entry_tid=41323' 2026-02-22T20:03:23.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:23.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=41556' = 'object_number=287, tag_tid=23, entry_tid=41323' 2026-02-22T20:03:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=284, tag_tid=23, entry_tid=41556' 2026-02-22T20:03:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:23.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:23.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=41556' 2026-02-22T20:03:23.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:23.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:23.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:23.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:23.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:23.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=41556' 2026-02-22T20:03:23.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:23.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:23.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41556 2026-02-22T20:03:23.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 41556 2026-02-22T20:03:23.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:23.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:23.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:24.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:24.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:24.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:24.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:24.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:24.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:24.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:24.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:24.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:24.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:24.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:24.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:24.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:24.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:24.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:24.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:24.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:24.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:24.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:24.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:24.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:24.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=41786], [object_number=285, tag_tid=23, entry_tid=41785], [object_number=284, tag_tid=23, entry_tid=41784], [object_number=287, tag_tid=23, entry_tid=41783]]], state=connected] 2026-02-22T20:03:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:24.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:24.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:24.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:24.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:24.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:24.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=41786], [object_number=285, tag_tid=23, entry_tid=41785], [object_number=284, tag_tid=23, entry_tid=41784], [object_number=287, tag_tid=23, entry_tid=41783]]], state=connected] 2026-02-22T20:03:24.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=286, tag_tid=23, entry_tid=41786' 2026-02-22T20:03:24.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=286, tag_tid=23, entry_tid=41786' 2026-02-22T20:03:24.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=41786' '!=' 'object_number=284, tag_tid=23, entry_tid=41556' 2026-02-22T20:03:24.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:24.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=41786' = 'object_number=284, tag_tid=23, entry_tid=41556' 2026-02-22T20:03:24.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=286, tag_tid=23, entry_tid=41786' 2026-02-22T20:03:24.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:24.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:24.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:24.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=41786' 2026-02-22T20:03:24.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:24.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:24.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:24.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:24.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:24.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=41786' 2026-02-22T20:03:24.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:24.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41786 2026-02-22T20:03:24.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 41786 2026-02-22T20:03:24.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:24.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:24.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:24.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:24.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:24.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:24.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:24.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:24.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:24.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:24.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:24.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:24.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:24.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:24.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:24.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:24.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:24.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:24.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:24.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:24.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:24.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:24.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:24.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:24.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:24.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:24.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:24.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:24.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:24.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:24.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=41979], [object_number=286, tag_tid=23, entry_tid=41978], [object_number=285, tag_tid=23, entry_tid=41977], [object_number=284, tag_tid=23, entry_tid=41976]]], state=connected] 2026-02-22T20:03:24.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:24.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:24.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:24.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:24.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:24.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:24.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=41979], [object_number=286, tag_tid=23, entry_tid=41978], [object_number=285, tag_tid=23, entry_tid=41977], [object_number=284, tag_tid=23, entry_tid=41976]]], state=connected] 2026-02-22T20:03:24.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=287, tag_tid=23, entry_tid=41979' 2026-02-22T20:03:24.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=287, tag_tid=23, entry_tid=41979' 2026-02-22T20:03:24.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=41979' '!=' 'object_number=286, tag_tid=23, entry_tid=41786' 2026-02-22T20:03:24.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:24.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=41979' = 'object_number=286, tag_tid=23, entry_tid=41786' 2026-02-22T20:03:24.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=287, tag_tid=23, entry_tid=41979' 2026-02-22T20:03:24.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:24.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:24.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:24.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=41979' 2026-02-22T20:03:24.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:24.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:24.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:24.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:24.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=41979' 2026-02-22T20:03:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41979 2026-02-22T20:03:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 41979 2026-02-22T20:03:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:24.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:24.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:24.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:24.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:24.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:24.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:24.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:24.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:24.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:24.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:24.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:24.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:24.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:24.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:24.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:24.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=285, tag_tid=23, entry_tid=42177], [object_number=284, tag_tid=23, entry_tid=42176], [object_number=287, tag_tid=23, entry_tid=42175], [object_number=286, tag_tid=23, entry_tid=42174]]], state=connected] 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:24.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:24.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:24.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:24.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:24.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:24.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:24.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=285, tag_tid=23, entry_tid=42177], [object_number=284, tag_tid=23, entry_tid=42176], [object_number=287, tag_tid=23, entry_tid=42175], [object_number=286, tag_tid=23, entry_tid=42174]]], state=connected] 2026-02-22T20:03:24.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:24.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=285, tag_tid=23, entry_tid=42177' 2026-02-22T20:03:24.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=285, tag_tid=23, entry_tid=42177' 2026-02-22T20:03:24.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=285, tag_tid=23, entry_tid=42177' '!=' 'object_number=287, tag_tid=23, entry_tid=41979' 2026-02-22T20:03:24.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:24.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=285, tag_tid=23, entry_tid=42177' = 'object_number=287, tag_tid=23, entry_tid=41979' 2026-02-22T20:03:24.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=285, tag_tid=23, entry_tid=42177' 2026-02-22T20:03:24.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:24.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:24.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:24.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=285, tag_tid=23, entry_tid=42177' 2026-02-22T20:03:24.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:24.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:24.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:24.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:24.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:24.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=285, tag_tid=23, entry_tid=42177' 2026-02-22T20:03:24.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:24.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42177 2026-02-22T20:03:24.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 42177 2026-02-22T20:03:24.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:24.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:24.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:25.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:25.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:25.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:25.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:25.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:25.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:25.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:25.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:25.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:25.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:25.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:25.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=42394], [object_number=285, tag_tid=23, entry_tid=42393], [object_number=284, tag_tid=23, entry_tid=42392], [object_number=287, tag_tid=23, entry_tid=42391]]], state=connected] 2026-02-22T20:03:25.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:25.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:25.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:25.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:25.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:25.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:25.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:25.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:25.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:25.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=42394], [object_number=285, tag_tid=23, entry_tid=42393], [object_number=284, tag_tid=23, entry_tid=42392], [object_number=287, tag_tid=23, entry_tid=42391]]], state=connected] 2026-02-22T20:03:25.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=286, tag_tid=23, entry_tid=42394' 2026-02-22T20:03:25.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=286, tag_tid=23, entry_tid=42394' 2026-02-22T20:03:25.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=42394' '!=' 'object_number=285, tag_tid=23, entry_tid=42177' 2026-02-22T20:03:25.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:25.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=42394' = 'object_number=285, tag_tid=23, entry_tid=42177' 2026-02-22T20:03:25.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=286, tag_tid=23, entry_tid=42394' 2026-02-22T20:03:25.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:25.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:25.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:25.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:25.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=42394' 2026-02-22T20:03:25.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:25.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:25.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:25.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:25.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:25.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:25.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:25.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=42394' 2026-02-22T20:03:25.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:25.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:25.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42394 2026-02-22T20:03:25.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 42394 2026-02-22T20:03:25.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:25.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:25.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:25.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:25.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:25.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:25.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:25.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:25.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:25.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:25.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:25.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:25.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:25.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:25.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:25.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:25.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:25.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:25.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:25.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:25.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:25.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:25.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:25.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:25.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:25.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:25.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=42596], [object_number=287, tag_tid=23, entry_tid=42595], [object_number=286, tag_tid=23, entry_tid=42594], [object_number=285, tag_tid=23, entry_tid=42593]]], state=connected] 2026-02-22T20:03:25.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:25.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:25.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:25.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:25.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:25.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:25.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:25.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:25.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:25.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=42596], [object_number=287, tag_tid=23, entry_tid=42595], [object_number=286, tag_tid=23, entry_tid=42594], [object_number=285, tag_tid=23, entry_tid=42593]]], state=connected] 2026-02-22T20:03:25.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=284, tag_tid=23, entry_tid=42596' 2026-02-22T20:03:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=284, tag_tid=23, entry_tid=42596' 2026-02-22T20:03:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=42596' '!=' 'object_number=286, tag_tid=23, entry_tid=42394' 2026-02-22T20:03:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=42596' = 'object_number=286, tag_tid=23, entry_tid=42394' 2026-02-22T20:03:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=284, tag_tid=23, entry_tid=42596' 2026-02-22T20:03:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=42596' 2026-02-22T20:03:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=42596' 2026-02-22T20:03:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42596 2026-02-22T20:03:25.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 42596 2026-02-22T20:03:25.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:25.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:25.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:25.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:25.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:25.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:25.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:25.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:25.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:25.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:25.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:25.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:25.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:25.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:25.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=42802], [object_number=285, tag_tid=23, entry_tid=42801], [object_number=284, tag_tid=23, entry_tid=42800], [object_number=287, tag_tid=23, entry_tid=42799]]], state=connected] 2026-02-22T20:03:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:25.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:25.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:25.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:25.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=42802], [object_number=285, tag_tid=23, entry_tid=42801], [object_number=284, tag_tid=23, entry_tid=42800], [object_number=287, tag_tid=23, entry_tid=42799]]], state=connected] 2026-02-22T20:03:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=286, tag_tid=23, entry_tid=42802' 2026-02-22T20:03:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=286, tag_tid=23, entry_tid=42802' 2026-02-22T20:03:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=42802' '!=' 'object_number=284, tag_tid=23, entry_tid=42596' 2026-02-22T20:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=42802' = 'object_number=284, tag_tid=23, entry_tid=42596' 2026-02-22T20:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=286, tag_tid=23, entry_tid=42802' 2026-02-22T20:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=42802' 2026-02-22T20:03:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:26.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:26.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:26.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=42802' 2026-02-22T20:03:26.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:26.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42802 2026-02-22T20:03:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 42802 2026-02-22T20:03:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:26.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:26.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:26.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:26.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:26.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:26.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:26.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:26.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:26.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:26.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:26.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:26.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:26.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:26.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:26.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:26.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:26.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:26.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:26.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:26.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:26.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:26.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:26.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:26.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:26.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=42999], [object_number=286, tag_tid=23, entry_tid=42998], [object_number=285, tag_tid=23, entry_tid=42997], [object_number=284, tag_tid=23, entry_tid=42996]]], state=connected] 2026-02-22T20:03:26.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:26.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:26.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:26.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:26.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:26.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:26.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:26.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:26.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=42999], [object_number=286, tag_tid=23, entry_tid=42998], [object_number=285, tag_tid=23, entry_tid=42997], [object_number=284, tag_tid=23, entry_tid=42996]]], state=connected] 2026-02-22T20:03:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=287, tag_tid=23, entry_tid=42999' 2026-02-22T20:03:26.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=287, tag_tid=23, entry_tid=42999' 2026-02-22T20:03:26.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=42999' '!=' 'object_number=286, tag_tid=23, entry_tid=42802' 2026-02-22T20:03:26.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:26.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=42999' = 'object_number=286, tag_tid=23, entry_tid=42802' 2026-02-22T20:03:26.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=287, tag_tid=23, entry_tid=42999' 2026-02-22T20:03:26.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:26.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:26.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:26.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=42999' 2026-02-22T20:03:26.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:26.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:26.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:26.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:26.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=42999' 2026-02-22T20:03:26.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:26.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=42999 2026-02-22T20:03:26.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 42999 2026-02-22T20:03:26.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:26.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:26.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:26.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:26.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:26.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:26.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:26.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:26.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:26.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:26.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:26.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:26.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:26.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:26.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:26.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:26.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:26.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:26.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:26.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:26.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:26.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:26.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:26.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:26.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:26.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:26.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:26.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:26.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:26.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:26.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:26.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:26.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:26.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:26.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:26.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:26.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:26.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:26.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:26.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:26.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:26.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:26.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:26.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:26.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:26.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:26.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:26.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:26.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:26.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:26.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:26.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:26.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=43222], [object_number=285, tag_tid=23, entry_tid=43221], [object_number=284, tag_tid=23, entry_tid=43220], [object_number=287, tag_tid=23, entry_tid=43219]]], state=connected] 2026-02-22T20:03:26.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:26.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:26.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:26.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:26.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:26.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:26.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:26.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:26.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:26.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=43222], [object_number=285, tag_tid=23, entry_tid=43221], [object_number=284, tag_tid=23, entry_tid=43220], [object_number=287, tag_tid=23, entry_tid=43219]]], state=connected] 2026-02-22T20:03:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=286, tag_tid=23, entry_tid=43222' 2026-02-22T20:03:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=286, tag_tid=23, entry_tid=43222' 2026-02-22T20:03:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=43222' '!=' 'object_number=287, tag_tid=23, entry_tid=42999' 2026-02-22T20:03:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=43222' = 'object_number=287, tag_tid=23, entry_tid=42999' 2026-02-22T20:03:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=286, tag_tid=23, entry_tid=43222' 2026-02-22T20:03:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:26.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=43222' 2026-02-22T20:03:26.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:26.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=43222' 2026-02-22T20:03:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:26.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43222 2026-02-22T20:03:26.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 43222 2026-02-22T20:03:26.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:26.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:26.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:26.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:26.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:26.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:26.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:27.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:27.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:27.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:27.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:27.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:27.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:27.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:27.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:27.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:27.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:27.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:27.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:27.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:27.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:27.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:27.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:27.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:27.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:27.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:27.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:27.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:27.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:27.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:27.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:27.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=43398], [object_number=285, tag_tid=23, entry_tid=43397], [object_number=284, tag_tid=23, entry_tid=43396], [object_number=287, tag_tid=23, entry_tid=43395]]], state=connected] 2026-02-22T20:03:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:27.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:27.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:27.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:27.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:27.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:27.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=286, tag_tid=23, entry_tid=43398], [object_number=285, tag_tid=23, entry_tid=43397], [object_number=284, tag_tid=23, entry_tid=43396], [object_number=287, tag_tid=23, entry_tid=43395]]], state=connected] 2026-02-22T20:03:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=286, tag_tid=23, entry_tid=43398' 2026-02-22T20:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=286, tag_tid=23, entry_tid=43398' 2026-02-22T20:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=43398' '!=' 'object_number=286, tag_tid=23, entry_tid=43222' 2026-02-22T20:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=286, tag_tid=23, entry_tid=43398' = 'object_number=286, tag_tid=23, entry_tid=43222' 2026-02-22T20:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=286, tag_tid=23, entry_tid=43398' 2026-02-22T20:03:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=43398' 2026-02-22T20:03:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:27.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=286, tag_tid=23, entry_tid=43398' 2026-02-22T20:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:27.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43398 2026-02-22T20:03:27.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 43398 2026-02-22T20:03:27.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:27.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:27.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:27.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:27.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:27.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:27.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:27.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:27.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:27.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:27.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:27.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:27.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:27.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:27.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:27.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=43623], [object_number=286, tag_tid=23, entry_tid=43622], [object_number=285, tag_tid=23, entry_tid=43621], [object_number=284, tag_tid=23, entry_tid=43620]]], state=connected] 2026-02-22T20:03:27.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:27.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:27.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:27.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:27.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:27.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:27.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:27.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:27.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=43623], [object_number=286, tag_tid=23, entry_tid=43622], [object_number=285, tag_tid=23, entry_tid=43621], [object_number=284, tag_tid=23, entry_tid=43620]]], state=connected] 2026-02-22T20:03:27.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=287, tag_tid=23, entry_tid=43623' 2026-02-22T20:03:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=287, tag_tid=23, entry_tid=43623' 2026-02-22T20:03:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=43623' '!=' 'object_number=286, tag_tid=23, entry_tid=43398' 2026-02-22T20:03:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=43623' = 'object_number=286, tag_tid=23, entry_tid=43398' 2026-02-22T20:03:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=287, tag_tid=23, entry_tid=43623' 2026-02-22T20:03:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:27.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:27.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=43623' 2026-02-22T20:03:27.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:27.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:27.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:27.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:27.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:27.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=43623' 2026-02-22T20:03:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:27.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43623 2026-02-22T20:03:27.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 43623 2026-02-22T20:03:27.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:27.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:27.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:27.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:27.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:27.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:27.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:27.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:27.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:27.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:28.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:28.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:28.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:28.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:28.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:28.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:28.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:28.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:28.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=43952], [object_number=287, tag_tid=23, entry_tid=43951], [object_number=286, tag_tid=23, entry_tid=43950], [object_number=285, tag_tid=23, entry_tid=43949]]], state=connected] 2026-02-22T20:03:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:28.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:28.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:28.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:28.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:28.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:28.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:28.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:28.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:28.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:28.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:28.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:28.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=43952], [object_number=287, tag_tid=23, entry_tid=43951], [object_number=286, tag_tid=23, entry_tid=43950], [object_number=285, tag_tid=23, entry_tid=43949]]], state=connected] 2026-02-22T20:03:28.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=284, tag_tid=23, entry_tid=43952' 2026-02-22T20:03:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=284, tag_tid=23, entry_tid=43952' 2026-02-22T20:03:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=43952' '!=' 'object_number=287, tag_tid=23, entry_tid=43623' 2026-02-22T20:03:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=43952' = 'object_number=287, tag_tid=23, entry_tid=43623' 2026-02-22T20:03:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=284, tag_tid=23, entry_tid=43952' 2026-02-22T20:03:28.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:28.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:28.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=43952' 2026-02-22T20:03:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:28.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:28.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:28.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=43952' 2026-02-22T20:03:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=43952 2026-02-22T20:03:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 43952 2026-02-22T20:03:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:28.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:28.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:28.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:28.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:28.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:28.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:28.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:28.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:28.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:28.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:28.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:28.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=44151], [object_number=286, tag_tid=23, entry_tid=44150], [object_number=285, tag_tid=23, entry_tid=44149], [object_number=284, tag_tid=23, entry_tid=44148]]], state=connected] 2026-02-22T20:03:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:28.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:28.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:28.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:28.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:28.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:28.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:28.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:28.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=287, tag_tid=23, entry_tid=44151], [object_number=286, tag_tid=23, entry_tid=44150], [object_number=285, tag_tid=23, entry_tid=44149], [object_number=284, tag_tid=23, entry_tid=44148]]], state=connected] 2026-02-22T20:03:28.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=287, tag_tid=23, entry_tid=44151' 2026-02-22T20:03:28.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=287, tag_tid=23, entry_tid=44151' 2026-02-22T20:03:28.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=44151' '!=' 'object_number=284, tag_tid=23, entry_tid=43952' 2026-02-22T20:03:28.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:28.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=287, tag_tid=23, entry_tid=44151' = 'object_number=284, tag_tid=23, entry_tid=43952' 2026-02-22T20:03:28.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=287, tag_tid=23, entry_tid=44151' 2026-02-22T20:03:28.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:28.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=44151' 2026-02-22T20:03:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:28.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=287, tag_tid=23, entry_tid=44151' 2026-02-22T20:03:28.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:28.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44151 2026-02-22T20:03:28.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 44151 2026-02-22T20:03:28.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:28.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:28.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:28.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:28.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:28.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:28.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:28.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:28.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:28.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:28.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:28.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:28.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:28.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:28.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:28.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:28.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:28.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:28.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:28.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:28.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:28.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:28.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:28.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=44348], [object_number=287, tag_tid=23, entry_tid=44347], [object_number=286, tag_tid=23, entry_tid=44346], [object_number=285, tag_tid=23, entry_tid=44345]]], state=connected] 2026-02-22T20:03:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:28.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:28.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:28.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:28.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:28.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:28.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:28.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:28.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:28.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:28.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:28.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:28.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=284, tag_tid=23, entry_tid=44348], [object_number=287, tag_tid=23, entry_tid=44347], [object_number=286, tag_tid=23, entry_tid=44346], [object_number=285, tag_tid=23, entry_tid=44345]]], state=connected] 2026-02-22T20:03:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:28.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=284, tag_tid=23, entry_tid=44348' 2026-02-22T20:03:28.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=284, tag_tid=23, entry_tid=44348' 2026-02-22T20:03:28.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=44348' '!=' 'object_number=287, tag_tid=23, entry_tid=44151' 2026-02-22T20:03:28.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:28.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=284, tag_tid=23, entry_tid=44348' = 'object_number=287, tag_tid=23, entry_tid=44151' 2026-02-22T20:03:28.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=284, tag_tid=23, entry_tid=44348' 2026-02-22T20:03:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:28.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=44348' 2026-02-22T20:03:28.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:28.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:28.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:28.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:28.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:28.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=284, tag_tid=23, entry_tid=44348' 2026-02-22T20:03:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44348 2026-02-22T20:03:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 44348 2026-02-22T20:03:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:29.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:29.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:29.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:29.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:29.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:29.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:29.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:29.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:29.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:29.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:29.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:29.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:29.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:29.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:29.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:29.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:29.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:29.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:29.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:29.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:29.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:29.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:29.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:29.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=285, tag_tid=23, entry_tid=44525], [object_number=288, tag_tid=23, entry_tid=44524], [object_number=287, tag_tid=23, entry_tid=44523], [object_number=286, tag_tid=23, entry_tid=44522]]], state=connected] 2026-02-22T20:03:29.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:29.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 71 2026-02-22T20:03:29.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:29.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:29.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:29.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=285, tag_tid=23, entry_tid=44525], [object_number=288, tag_tid=23, entry_tid=44524], [object_number=287, tag_tid=23, entry_tid=44523], [object_number=286, tag_tid=23, entry_tid=44522]]], state=connected] 2026-02-22T20:03:29.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=285, tag_tid=23, entry_tid=44525' 2026-02-22T20:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=285, tag_tid=23, entry_tid=44525' 2026-02-22T20:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=285, tag_tid=23, entry_tid=44525' '!=' 'object_number=284, tag_tid=23, entry_tid=44348' 2026-02-22T20:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=285, tag_tid=23, entry_tid=44525' = 'object_number=284, tag_tid=23, entry_tid=44348' 2026-02-22T20:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=285, tag_tid=23, entry_tid=44525' 2026-02-22T20:03:29.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:29.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:29.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:29.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=285, tag_tid=23, entry_tid=44525' 2026-02-22T20:03:29.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:29.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=285, tag_tid=23, entry_tid=44525' 2026-02-22T20:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44525 2026-02-22T20:03:29.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 44525 2026-02-22T20:03:29.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:29.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:29.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:29.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:29.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:29.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:29.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:29.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:29.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:29.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:29.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:29.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:29.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:29.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:29.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=290, tag_tid=23, entry_tid=44690], [object_number=289, tag_tid=23, entry_tid=44689], [object_number=288, tag_tid=23, entry_tid=44688], [object_number=291, tag_tid=23, entry_tid=44687]]], state=connected] 2026-02-22T20:03:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:29.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:29.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:29.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:29.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:29.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:29.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:29.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:29.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:29.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:29.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=290, tag_tid=23, entry_tid=44690], [object_number=289, tag_tid=23, entry_tid=44689], [object_number=288, tag_tid=23, entry_tid=44688], [object_number=291, tag_tid=23, entry_tid=44687]]], state=connected] 2026-02-22T20:03:29.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=290, tag_tid=23, entry_tid=44690' 2026-02-22T20:03:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=290, tag_tid=23, entry_tid=44690' 2026-02-22T20:03:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=290, tag_tid=23, entry_tid=44690' '!=' 'object_number=285, tag_tid=23, entry_tid=44525' 2026-02-22T20:03:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=290, tag_tid=23, entry_tid=44690' = 'object_number=285, tag_tid=23, entry_tid=44525' 2026-02-22T20:03:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=290, tag_tid=23, entry_tid=44690' 2026-02-22T20:03:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:29.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=290, tag_tid=23, entry_tid=44690' 2026-02-22T20:03:29.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:29.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:29.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:29.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:29.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:29.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=290, tag_tid=23, entry_tid=44690' 2026-02-22T20:03:29.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:29.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44690 2026-02-22T20:03:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 44690 2026-02-22T20:03:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:29.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:29.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:29.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:29.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:29.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:29.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:29.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:29.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:29.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:29.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:29.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:29.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=288, tag_tid=23, entry_tid=44880], [object_number=291, tag_tid=23, entry_tid=44879], [object_number=290, tag_tid=23, entry_tid=44878], [object_number=289, tag_tid=23, entry_tid=44877]]], state=connected] 2026-02-22T20:03:29.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:29.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:29.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:29.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:29.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:29.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=288, tag_tid=23, entry_tid=44880], [object_number=291, tag_tid=23, entry_tid=44879], [object_number=290, tag_tid=23, entry_tid=44878], [object_number=289, tag_tid=23, entry_tid=44877]]], state=connected] 2026-02-22T20:03:29.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:29.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=288, tag_tid=23, entry_tid=44880' 2026-02-22T20:03:29.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=288, tag_tid=23, entry_tid=44880' 2026-02-22T20:03:29.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=288, tag_tid=23, entry_tid=44880' '!=' 'object_number=290, tag_tid=23, entry_tid=44690' 2026-02-22T20:03:29.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:29.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=288, tag_tid=23, entry_tid=44880' = 'object_number=290, tag_tid=23, entry_tid=44690' 2026-02-22T20:03:29.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=288, tag_tid=23, entry_tid=44880' 2026-02-22T20:03:29.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:29.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:29.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:29.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=288, tag_tid=23, entry_tid=44880' 2026-02-22T20:03:29.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:29.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:29.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:29.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:29.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:29.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=288, tag_tid=23, entry_tid=44880' 2026-02-22T20:03:29.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:29.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:29.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44880 2026-02-22T20:03:29.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 44880 2026-02-22T20:03:29.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:29.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:29.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:30.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:30.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:30.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:30.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:30.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:30.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:30.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:30.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:30.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:30.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:30.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:30.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:30.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:30.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:30.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:30.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:30.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:30.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:30.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:30.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:30.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:30.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:30.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:30.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=290, tag_tid=23, entry_tid=45086], [object_number=289, tag_tid=23, entry_tid=45085], [object_number=288, tag_tid=23, entry_tid=45084], [object_number=291, tag_tid=23, entry_tid=45083]]], state=connected] 2026-02-22T20:03:30.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:30.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:30.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:30.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:30.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=290, tag_tid=23, entry_tid=45086], [object_number=289, tag_tid=23, entry_tid=45085], [object_number=288, tag_tid=23, entry_tid=45084], [object_number=291, tag_tid=23, entry_tid=45083]]], state=connected] 2026-02-22T20:03:30.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=290, tag_tid=23, entry_tid=45086' 2026-02-22T20:03:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=290, tag_tid=23, entry_tid=45086' 2026-02-22T20:03:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=290, tag_tid=23, entry_tid=45086' '!=' 'object_number=288, tag_tid=23, entry_tid=44880' 2026-02-22T20:03:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=290, tag_tid=23, entry_tid=45086' = 'object_number=288, tag_tid=23, entry_tid=44880' 2026-02-22T20:03:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=290, tag_tid=23, entry_tid=45086' 2026-02-22T20:03:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:30.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=290, tag_tid=23, entry_tid=45086' 2026-02-22T20:03:30.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:30.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:30.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:30.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:30.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:30.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:30.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:30.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=290, tag_tid=23, entry_tid=45086' 2026-02-22T20:03:30.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:30.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:30.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45086 2026-02-22T20:03:30.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 45086 2026-02-22T20:03:30.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:30.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:30.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:30.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:30.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:30.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:30.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:30.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:30.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:30.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:30.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:30.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:30.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:30.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:30.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:30.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=288, tag_tid=23, entry_tid=45304], [object_number=291, tag_tid=23, entry_tid=45303], [object_number=290, tag_tid=23, entry_tid=45302], [object_number=289, tag_tid=23, entry_tid=45301]]], state=connected] 2026-02-22T20:03:30.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:30.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:30.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:30.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:30.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:30.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:30.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:30.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:30.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:30.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:30.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:30.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:30.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:30.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:30.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=288, tag_tid=23, entry_tid=45304], [object_number=291, tag_tid=23, entry_tid=45303], [object_number=290, tag_tid=23, entry_tid=45302], [object_number=289, tag_tid=23, entry_tid=45301]]], state=connected] 2026-02-22T20:03:30.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:30.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=288, tag_tid=23, entry_tid=45304' 2026-02-22T20:03:30.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=288, tag_tid=23, entry_tid=45304' 2026-02-22T20:03:30.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=288, tag_tid=23, entry_tid=45304' '!=' 'object_number=290, tag_tid=23, entry_tid=45086' 2026-02-22T20:03:30.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:30.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=288, tag_tid=23, entry_tid=45304' = 'object_number=290, tag_tid=23, entry_tid=45086' 2026-02-22T20:03:30.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=288, tag_tid=23, entry_tid=45304' 2026-02-22T20:03:30.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:30.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:30.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:30.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:30.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=288, tag_tid=23, entry_tid=45304' 2026-02-22T20:03:30.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:30.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:30.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:30.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:30.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:30.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=288, tag_tid=23, entry_tid=45304' 2026-02-22T20:03:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45304 2026-02-22T20:03:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 45304 2026-02-22T20:03:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:30.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:30.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:30.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:31.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:31.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:31.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:31.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:31.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:31.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:31.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:31.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=289, tag_tid=23, entry_tid=45517], [object_number=288, tag_tid=23, entry_tid=45516], [object_number=291, tag_tid=23, entry_tid=45515], [object_number=290, tag_tid=23, entry_tid=45514]]], state=connected] 2026-02-22T20:03:31.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:31.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:31.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:31.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=289, tag_tid=23, entry_tid=45517], [object_number=288, tag_tid=23, entry_tid=45516], [object_number=291, tag_tid=23, entry_tid=45515], [object_number=290, tag_tid=23, entry_tid=45514]]], state=connected] 2026-02-22T20:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=289, tag_tid=23, entry_tid=45517' 2026-02-22T20:03:31.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=289, tag_tid=23, entry_tid=45517' 2026-02-22T20:03:31.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=289, tag_tid=23, entry_tid=45517' '!=' 'object_number=288, tag_tid=23, entry_tid=45304' 2026-02-22T20:03:31.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:31.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=289, tag_tid=23, entry_tid=45517' = 'object_number=288, tag_tid=23, entry_tid=45304' 2026-02-22T20:03:31.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=289, tag_tid=23, entry_tid=45517' 2026-02-22T20:03:31.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:31.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:31.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:31.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=23, entry_tid=45517' 2026-02-22T20:03:31.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:31.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:31.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:31.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:31.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:31.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=23, entry_tid=45517' 2026-02-22T20:03:31.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:31.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45517 2026-02-22T20:03:31.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 45517 2026-02-22T20:03:31.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:31.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:31.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:31.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:31.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:31.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:31.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:31.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:31.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:31.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:31.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:31.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:31.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:31.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=290, tag_tid=23, entry_tid=45762], [object_number=289, tag_tid=23, entry_tid=45761], [object_number=288, tag_tid=23, entry_tid=45760], [object_number=291, tag_tid=23, entry_tid=45759]]], state=connected] 2026-02-22T20:03:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:31.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:31.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:31.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:31.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:31.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:31.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=290, tag_tid=23, entry_tid=45762], [object_number=289, tag_tid=23, entry_tid=45761], [object_number=288, tag_tid=23, entry_tid=45760], [object_number=291, tag_tid=23, entry_tid=45759]]], state=connected] 2026-02-22T20:03:31.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=290, tag_tid=23, entry_tid=45762' 2026-02-22T20:03:31.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=290, tag_tid=23, entry_tid=45762' 2026-02-22T20:03:31.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=290, tag_tid=23, entry_tid=45762' '!=' 'object_number=289, tag_tid=23, entry_tid=45517' 2026-02-22T20:03:31.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:31.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=290, tag_tid=23, entry_tid=45762' = 'object_number=289, tag_tid=23, entry_tid=45517' 2026-02-22T20:03:31.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=290, tag_tid=23, entry_tid=45762' 2026-02-22T20:03:31.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:31.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:31.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:31.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=290, tag_tid=23, entry_tid=45762' 2026-02-22T20:03:31.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:31.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:31.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:31.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:31.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:31.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=290, tag_tid=23, entry_tid=45762' 2026-02-22T20:03:31.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:31.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45762 2026-02-22T20:03:31.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 45762 2026-02-22T20:03:31.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:31.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:31.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:31.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:31.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:31.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:31.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:31.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:31.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:31.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:31.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:31.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:31.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:31.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:31.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:31.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:31.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:31.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=288, tag_tid=23, entry_tid=45960], [object_number=291, tag_tid=23, entry_tid=45959], [object_number=290, tag_tid=23, entry_tid=45958], [object_number=289, tag_tid=23, entry_tid=45957]]], state=connected] 2026-02-22T20:03:31.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:31.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:31.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:31.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:31.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:31.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:31.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=288, tag_tid=23, entry_tid=45960], [object_number=291, tag_tid=23, entry_tid=45959], [object_number=290, tag_tid=23, entry_tid=45958], [object_number=289, tag_tid=23, entry_tid=45957]]], state=connected] 2026-02-22T20:03:31.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:31.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=288, tag_tid=23, entry_tid=45960' 2026-02-22T20:03:31.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=288, tag_tid=23, entry_tid=45960' 2026-02-22T20:03:31.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=288, tag_tid=23, entry_tid=45960' '!=' 'object_number=290, tag_tid=23, entry_tid=45762' 2026-02-22T20:03:31.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:31.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=288, tag_tid=23, entry_tid=45960' = 'object_number=290, tag_tid=23, entry_tid=45762' 2026-02-22T20:03:31.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=288, tag_tid=23, entry_tid=45960' 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:31.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=288, tag_tid=23, entry_tid=45960' 2026-02-22T20:03:31.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:31.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:31.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=288, tag_tid=23, entry_tid=45960' 2026-02-22T20:03:31.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:31.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:31.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45960 2026-02-22T20:03:31.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 45960 2026-02-22T20:03:31.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:31.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:31.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:32.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:32.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:32.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:32.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:32.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:32.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:32.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:32.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:32.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:32.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:32.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:32.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=290, tag_tid=23, entry_tid=46162], [object_number=289, tag_tid=23, entry_tid=46161], [object_number=288, tag_tid=23, entry_tid=46160], [object_number=291, tag_tid=23, entry_tid=46159]]], state=connected] 2026-02-22T20:03:32.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:32.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:32.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:32.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:32.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:32.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=290, tag_tid=23, entry_tid=46162], [object_number=289, tag_tid=23, entry_tid=46161], [object_number=288, tag_tid=23, entry_tid=46160], [object_number=291, tag_tid=23, entry_tid=46159]]], state=connected] 2026-02-22T20:03:32.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=290, tag_tid=23, entry_tid=46162' 2026-02-22T20:03:32.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=290, tag_tid=23, entry_tid=46162' 2026-02-22T20:03:32.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=290, tag_tid=23, entry_tid=46162' '!=' 'object_number=288, tag_tid=23, entry_tid=45960' 2026-02-22T20:03:32.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:32.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=290, tag_tid=23, entry_tid=46162' = 'object_number=288, tag_tid=23, entry_tid=45960' 2026-02-22T20:03:32.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=290, tag_tid=23, entry_tid=46162' 2026-02-22T20:03:32.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:32.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:32.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:32.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=290, tag_tid=23, entry_tid=46162' 2026-02-22T20:03:32.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:32.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:32.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:32.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=290, tag_tid=23, entry_tid=46162' 2026-02-22T20:03:32.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:32.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46162 2026-02-22T20:03:32.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 46162 2026-02-22T20:03:32.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:32.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:32.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:32.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:32.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:32.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:32.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:32.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:32.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:32.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:32.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:32.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:32.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:32.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:32.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:32.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:32.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:32.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:32.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:32.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:32.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:32.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:32.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:32.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:32.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:32.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:32.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:32.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:32.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:32.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:32.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:32.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:32.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:32.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=291, tag_tid=23, entry_tid=46367], [object_number=290, tag_tid=23, entry_tid=46366], [object_number=289, tag_tid=23, entry_tid=46365], [object_number=288, tag_tid=23, entry_tid=46364]]], state=connected] 2026-02-22T20:03:32.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=291, tag_tid=23, entry_tid=46367], [object_number=290, tag_tid=23, entry_tid=46366], [object_number=289, tag_tid=23, entry_tid=46365], [object_number=288, tag_tid=23, entry_tid=46364]]], state=connected] 2026-02-22T20:03:32.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=291, tag_tid=23, entry_tid=46367' 2026-02-22T20:03:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=291, tag_tid=23, entry_tid=46367' 2026-02-22T20:03:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=291, tag_tid=23, entry_tid=46367' '!=' 'object_number=290, tag_tid=23, entry_tid=46162' 2026-02-22T20:03:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=291, tag_tid=23, entry_tid=46367' = 'object_number=290, tag_tid=23, entry_tid=46162' 2026-02-22T20:03:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=291, tag_tid=23, entry_tid=46367' 2026-02-22T20:03:32.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:32.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:32.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=291, tag_tid=23, entry_tid=46367' 2026-02-22T20:03:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:32.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:32.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:32.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=291, tag_tid=23, entry_tid=46367' 2026-02-22T20:03:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:32.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46367 2026-02-22T20:03:32.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 46367 2026-02-22T20:03:32.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:32.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:32.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:32.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:32.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:32.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:32.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:32.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:32.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:32.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:32.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:32.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:32.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:32.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:32.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:32.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:32.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:32.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:32.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:32.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:32.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:32.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:32.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:32.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:32.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:32.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:32.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:32.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:32.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:32.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:32.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:32.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=291, tag_tid=23, entry_tid=46571], [object_number=290, tag_tid=23, entry_tid=46570], [object_number=289, tag_tid=23, entry_tid=46569], [object_number=288, tag_tid=23, entry_tid=46568]]], state=connected] 2026-02-22T20:03:32.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:32.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:32.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:32.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:32.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:32.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:32.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:32.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:32.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:32.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=291, tag_tid=23, entry_tid=46571], [object_number=290, tag_tid=23, entry_tid=46570], [object_number=289, tag_tid=23, entry_tid=46569], [object_number=288, tag_tid=23, entry_tid=46568]]], state=connected] 2026-02-22T20:03:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:32.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=291, tag_tid=23, entry_tid=46571' 2026-02-22T20:03:32.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=291, tag_tid=23, entry_tid=46571' 2026-02-22T20:03:32.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=291, tag_tid=23, entry_tid=46571' '!=' 'object_number=291, tag_tid=23, entry_tid=46367' 2026-02-22T20:03:32.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:32.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=291, tag_tid=23, entry_tid=46571' = 'object_number=291, tag_tid=23, entry_tid=46367' 2026-02-22T20:03:32.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=291, tag_tid=23, entry_tid=46571' 2026-02-22T20:03:32.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:32.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:32.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:32.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=291, tag_tid=23, entry_tid=46571' 2026-02-22T20:03:32.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:32.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:32.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:32.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:32.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:32.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=291, tag_tid=23, entry_tid=46571' 2026-02-22T20:03:32.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:32.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:32.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46571 2026-02-22T20:03:32.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 46571 2026-02-22T20:03:32.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:32.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:32.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:33.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:33.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:33.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:33.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:33.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:33.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:33.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:33.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:33.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:33.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:33.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:33.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:33.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:33.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=288, tag_tid=23, entry_tid=46768], [object_number=291, tag_tid=23, entry_tid=46767], [object_number=290, tag_tid=23, entry_tid=46766], [object_number=289, tag_tid=23, entry_tid=46765]]], state=connected] 2026-02-22T20:03:33.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:33.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:33.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:33.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:33.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:33.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:33.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:33.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:33.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:33.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=288, tag_tid=23, entry_tid=46768], [object_number=291, tag_tid=23, entry_tid=46767], [object_number=290, tag_tid=23, entry_tid=46766], [object_number=289, tag_tid=23, entry_tid=46765]]], state=connected] 2026-02-22T20:03:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=288, tag_tid=23, entry_tid=46768' 2026-02-22T20:03:33.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=288, tag_tid=23, entry_tid=46768' 2026-02-22T20:03:33.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=288, tag_tid=23, entry_tid=46768' '!=' 'object_number=291, tag_tid=23, entry_tid=46571' 2026-02-22T20:03:33.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:33.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=288, tag_tid=23, entry_tid=46768' = 'object_number=291, tag_tid=23, entry_tid=46571' 2026-02-22T20:03:33.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=288, tag_tid=23, entry_tid=46768' 2026-02-22T20:03:33.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:33.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:33.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:33.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:33.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=288, tag_tid=23, entry_tid=46768' 2026-02-22T20:03:33.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:33.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:33.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:33.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:33.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:33.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:33.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:33.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=288, tag_tid=23, entry_tid=46768' 2026-02-22T20:03:33.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:33.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:33.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46768 2026-02-22T20:03:33.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 46768 2026-02-22T20:03:33.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:33.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:33.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:33.320 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:03:33.323+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2d79d520 2026-02-22T20:03:33.321 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:03:33.323+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e096270 2026-02-22T20:03:33.321 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:03:33.323+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0964e0 2026-02-22T20:03:33.322 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:03:33.323+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e096680 2026-02-22T20:03:33.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:33.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:33.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:33.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:33.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:33.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:33.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:33.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:33.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:33.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:33.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:33.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:33.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:33.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:33.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:33.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:33.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:33.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:33.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:33.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:33.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:33.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:33.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:33.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:33.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:33.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:33.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:33.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:33.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:33.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:33.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:33.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:33.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:33.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:33.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=289, tag_tid=23, entry_tid=46965], [object_number=288, tag_tid=23, entry_tid=46964], [object_number=291, tag_tid=23, entry_tid=46963], [object_number=290, tag_tid=23, entry_tid=46962]]], state=connected] 2026-02-22T20:03:33.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:33.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:33.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:33.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:33.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:33.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:33.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:33.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:33.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:33.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=289, tag_tid=23, entry_tid=46965], [object_number=288, tag_tid=23, entry_tid=46964], [object_number=291, tag_tid=23, entry_tid=46963], [object_number=290, tag_tid=23, entry_tid=46962]]], state=connected] 2026-02-22T20:03:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=289, tag_tid=23, entry_tid=46965' 2026-02-22T20:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=289, tag_tid=23, entry_tid=46965' 2026-02-22T20:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=289, tag_tid=23, entry_tid=46965' '!=' 'object_number=288, tag_tid=23, entry_tid=46768' 2026-02-22T20:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=289, tag_tid=23, entry_tid=46965' = 'object_number=288, tag_tid=23, entry_tid=46768' 2026-02-22T20:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=289, tag_tid=23, entry_tid=46965' 2026-02-22T20:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=23, entry_tid=46965' 2026-02-22T20:03:33.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:33.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:33.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:33.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:33.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:33.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:33.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=23, entry_tid=46965' 2026-02-22T20:03:33.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:33.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:33.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=46965 2026-02-22T20:03:33.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 46965 2026-02-22T20:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:33.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:33.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:33.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:33.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:33.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:33.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:33.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:33.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:33.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:33.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:33.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:33.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:33.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:33.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:33.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:33.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:33.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:33.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:33.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:33.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:33.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:33.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:33.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:33.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:33.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:33.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:33.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002], [object_number=289, tag_tid=23, entry_tid=47001], [object_number=288, tag_tid=23, entry_tid=47000]]], state=connected] 2026-02-22T20:03:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:34.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:34.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:34.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:34.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:34.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:34.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:34.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:34.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:34.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002], [object_number=289, tag_tid=23, entry_tid=47001], [object_number=288, tag_tid=23, entry_tid=47000]]], state=connected] 2026-02-22T20:03:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=291, tag_tid=23, entry_tid=47003' 2026-02-22T20:03:34.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=291, tag_tid=23, entry_tid=47003' 2026-02-22T20:03:34.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=291, tag_tid=23, entry_tid=47003' '!=' 'object_number=289, tag_tid=23, entry_tid=46965' 2026-02-22T20:03:34.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:03:34.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=291, tag_tid=23, entry_tid=47003' = 'object_number=289, tag_tid=23, entry_tid=46965' 2026-02-22T20:03:34.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=291, tag_tid=23, entry_tid=47003' 2026-02-22T20:03:34.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:34.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:34.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:34.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=24 2026-02-22T20:03:34.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=291, tag_tid=23, entry_tid=47003' 2026-02-22T20:03:34.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:03:34.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:34.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=23 2026-02-22T20:03:34.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:34.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:34.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:34.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:03:34.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=291, tag_tid=23, entry_tid=47003' 2026-02-22T20:03:34.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:03:34.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:03:34.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=47003 2026-02-22T20:03:34.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 24 = 23 -a 1 -le 47003 2026-02-22T20:03:34.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:03:34.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:34.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:03:34.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:03:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:03:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:03:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:03:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:03:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:03:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:03:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:03:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:03:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:03:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:03:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:03:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:03:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:03:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:03:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:03:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:03:34.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:34.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:34.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:34.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:03:34.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:34.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:03:34.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:34.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:34.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:34.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:03:34.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:34.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:34.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:34.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:03:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:34.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:34.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:34.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:34.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:34.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:34.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:03:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:03:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:03:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:03:34.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:03:34.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:03:34.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:34.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:03:34.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:34.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:03:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 72 2026-02-22T20:03:34.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 72 2026-02-22T20:03:34.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:03:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=289, tag_tid=24, entry_tid=1], [object_number=288, tag_tid=24, entry_tid=0], [object_number=291, tag_tid=23, entry_tid=47003], [object_number=290, tag_tid=23, entry_tid=47002]]], state=connected] 2026-02-22T20:03:34.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:03:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=289, tag_tid=24, entry_tid=1' -a 'object_number=289, tag_tid=24, entry_tid=1' = 'object_number=289, tag_tid=24, entry_tid=1' 2026-02-22T20:03:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2026-02-22T20:03:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap8 2026-02-22T20:03:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:03:34.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:03:34.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:03:35.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2026-02-22T20:03:35.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T20:03:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:03:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:03:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 15 2026-02-22T20:03:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T20:03:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T20:03:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:03:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:03:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:03:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:03:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T20:03:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:01:04 2026 2026-02-22T20:03:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:02:44 2026 2026-02-22T20:03:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:03:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T20:03:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:03:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:03:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:03:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:03:35.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:35.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n snap7-clean ']' 2026-02-22T20:03:35.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap7-clean 2026-02-22T20:03:35.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:35.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:35.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap7-clean 2026-02-22T20:03:35.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:03:35.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:03:35.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:35.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:35.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:03:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:03:35.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:36.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:03:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:36.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:36.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:36.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap8 2026-02-22T20:03:36.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:36.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:36.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap8 2026-02-22T20:03:36.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:03:36.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:03:36.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:36.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:36.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:03:37.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:03:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:38.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:03:38.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:38.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:03:38.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:38.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap8-clean 2026-02-22T20:03:38.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap8-clean 2026-02-22T20:03:38.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:03:38.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:38.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:38.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap8-clean 2026-02-22T20:03:38.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T20:03:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:03:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T20:03:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:03:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:03:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=6 2026-02-22T20:03:38.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:03:38.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T20:03:39.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 34092032 bytes 2026-02-22T20:03:40.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 65220608 bytes 2026-02-22T20:03:41.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 132377600 bytes 2026-02-22T20:03:41.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 132996096 bytes 2026-02-22T20:03:42.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 170299392 bytes 2026-02-22T20:03:42.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 198966272 bytes 2026-02-22T20:03:43.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 215704576 bytes 2026-02-22T20:03:43.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 231188480 bytes 2026-02-22T20:03:43.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 255395840 bytes 2026-02-22T20:03:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 282005504 bytes 2026-02-22T20:03:44.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:03:44.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:03:44.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:03:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:03:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:03:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:03:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:44.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:03:45.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:03:45.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:45.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:45.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:45.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:45.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:03:45.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:03:45.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:03:45.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:03:45.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:03:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T20:03:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:03:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T20:03:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":8650427.199999999,"entries_behind_primary":433,"entries_per_second":543.37,"non_primary_position":{"entry_tid":46571,"object_number":291,"tag_tid":23},"primary_position":{"entry_tid":1,"object_number":289,"tag_tid":24},"seconds_until_synced":0} 2026-02-22T20:03:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.0 on trial066 2026-02-22T20:03:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:03:33 2026-02-22T20:03:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T20:03:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T20:03:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T20:03:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T20:03:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:03:33' 2026-02-22T20:03:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:03:45.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T20:03:45.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T20:03:45.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":8650427.199999999,"entries_behind_primary":433,"entries_per_second":543.37,"non_primary_position":{"entry_tid":46571,"object_number":291,"tag_tid":23},"primary_position":{"entry_tid":1,"object_number":289,"tag_tid":24},"seconds_until_synced":0} 2026-02-22T20:03:45.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:03:45.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:03:45.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T20:03:45.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:03:45.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:45.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.0 on trial066 2026-02-22T20:03:45.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T20:03:45.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:03:33' 2026-02-22T20:03:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:03:33' 2026-02-22T20:03:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:03:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 20:03:33' 2026-02-22T20:03:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:03:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:03:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142024257mirror.0trial06619.2.3-689-gf90f4eaetrueOK42134266mirror.3trial06619.2.3-689-gf90f4eaefalseOK43154318mirror.1trial06619.2.3-689-gf90f4eaefalseOK43894391mirror.2trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":8650427.199999999,"entries_behind_primary":433,"entries_per_second":543.37,"non_primary_position":{"entry_tid":46571,"object_number":291,"tag_tid":23},"primary_position":{"entry_tid":1,"object_number":289,"tag_tid":24},"seconds_until_synced":0}42024257mirror.0trial0662026-02-22 20:03:33cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 20:03:33' 2026-02-22T20:03:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T20:03:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:03:33' 2026-02-22T20:03:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T20:03:45.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:03:45.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T20:03:45.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:03:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:03:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap9 2026-02-22T20:03:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap9 2026-02-22T20:03:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:03:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:03:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:03:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap9 2026-02-22T20:03:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:03:53.248 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2026-02-22T20:03:53.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2026-02-22T20:03:53.249 INFO:tasks.rbd_mirror_thrash:waiting for 5.058587252172199 secs before reviving daemons 2026-02-22T20:03:53.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:03:53.236+0000 7f277db8d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 19978) UID: 1000 2026-02-22T20:03:55.083 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3525759794985579) > weight (0.25) 2026-02-22T20:03:55.083 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3176101077492637) > weight (0.25) 2026-02-22T20:03:55.083 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.44054716932970917) > weight (0.25) 2026-02-22T20:03:55.083 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6719363772741647) > weight (0.25) 2026-02-22T20:03:55.083 INFO:tasks.rbd_mirror_thrash:waiting for 84.56108172072443 secs before thrashing 2026-02-22T20:03:58.308 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2026-02-22T20:03:58.308 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:04:04.409 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2026-02-22T20:04:04.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-22T20:04:04.409 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-22T20:04:04.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-22T20:04:04.412 INFO:tasks.rbd_mirror_thrash:waiting for 82.31236328160648 secs before thrashing 2026-02-22T20:04:15.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:04:15.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:15.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:15.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:15.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:04:15.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:15.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:04:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:04:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:04:16.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:04:16.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:16.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:16.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:16.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:04:16.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:04:16.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:04:16.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:04:16.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:04:16.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:04:16.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:16.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:16.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:04:16.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:04:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:04:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:04:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:16.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:04:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:04:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:16.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:16.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:16.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:16.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:16.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:16.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:16.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:16.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:04:16.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T20:04:16.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:16.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:16.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:16.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:16.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:16.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:16.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:16.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:16.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:16.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:04:16.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:04:16.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:16.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:16.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:16.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:16.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T20:04:17.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:17.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:17.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:17.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:17.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:17.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:17.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:04:17.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:04:17.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:17.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:17.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:17.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:17.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T20:04:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:17.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:17.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:17.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 1 2026-02-22T20:04:18.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:18.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:04:18.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:18.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:18.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:18.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:18.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:04:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:04:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:18.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:18.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:18.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:18.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:18.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:18.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:18.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:18.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:18.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T20:04:18.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:04:18.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:18.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:18.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:18.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:18.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:18.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:18.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:18.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:18.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:18.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T20:04:18.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:18.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:18.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:18.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:18.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:04:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:04:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:18.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:18.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:18.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:18.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T20:04:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:18.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:18.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T20:04:18.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:04:18.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:04:18.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:04:18.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:18.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:18.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:18.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:18.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T20:04:18.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:18.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:18.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:18.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:18.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 2 2026-02-22T20:04:20.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:20.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:04:20.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:20.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:20.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:20.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:20.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:20.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:20.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:04:20.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:04:20.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:20.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:20.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:20.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:20.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:20.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:20.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:20.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:20.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:20.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T20:04:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:04:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:20.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:20.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:20.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:20.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T20:04:20.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:20.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:20.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:20.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:20.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T20:04:20.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:04:20.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:20.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:20.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:04:20.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:04:20.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:04:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:20.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T20:04:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T20:04:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:04:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:04:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:20.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:20.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:20.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:20.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T20:04:20.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:20.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:20.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:20.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:20.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 4 2026-02-22T20:04:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:24.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:04:24.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:24.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:24.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:24.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:24.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:24.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:24.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:04:24.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:04:24.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:24.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:24.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:24.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:24.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:24.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:24.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:24.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T20:04:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:24.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:24.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:24.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:04:24.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:04:24.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:24.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:24.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:24.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:24.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T20:04:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:24.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:24.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:24.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:24.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:24.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:24.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:24.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:24.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:24.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:24.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:04:24.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:04:24.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:24.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:24.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:24.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:24.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T20:04:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:24.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:24.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:24.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T20:04:24.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:04:24.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:24.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:24.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:24.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:24.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:24.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:24.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:04:24.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:24.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:24.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:24.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:24.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:24.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:24.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:24.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:04:24.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:04:24.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:24.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:24.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:24.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:24.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T20:04:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:24.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:24.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 8 2026-02-22T20:04:32.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:32.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:04:32.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:32.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:32.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:32.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:32.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:04:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:04:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:32.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:32.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:32.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:32.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:32.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:32.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:32.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T20:04:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:04:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:04:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:33.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:33.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:33.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:33.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T20:04:33.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:33.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:33.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:33.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:33.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:33.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:33.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:33.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:33.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:04:33.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:04:33.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:33.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:33.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:33.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T20:04:33.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:33.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:04:33.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T20:04:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:04:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:04:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:04:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:04:33.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:33.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:33.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:33.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:33.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T20:04:33.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:04:33.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:04:33.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:04:33.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:04:33.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:04:33.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:04:33.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:04:33.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:33.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:04:33.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:04:33.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:33.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:33.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:33.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:33.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:33.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:33.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:33.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:33.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:33.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:33.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:33.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:33.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:33.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:33.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:33.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:33.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:33.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:33.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:33.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:33.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:33.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:33.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:33.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:33.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:33.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:33.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:33.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:33.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:33.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:33.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:33.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:33.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:33.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:33.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:33.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:33.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:33.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:33.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:33.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:33.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:33.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:33.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:33.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:33.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:33.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:33.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:33.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:33.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:33.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:33.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:33.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:33.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:33.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:33.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:33.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:33.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:33.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:33.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:33.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:33.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:33.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:33.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:33.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:33.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:33.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:33.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:33.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:33.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:33.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:33.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:33.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=297, tag_tid=26, entry_tid=7561], [object_number=296, tag_tid=26, entry_tid=7560], [object_number=299, tag_tid=26, entry_tid=7559], [object_number=298, tag_tid=26, entry_tid=7558]]], state=connected] 2026-02-22T20:04:33.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:33.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:33.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:33.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:33.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=297, tag_tid=26, entry_tid=7561], [object_number=296, tag_tid=26, entry_tid=7560], [object_number=299, tag_tid=26, entry_tid=7559], [object_number=298, tag_tid=26, entry_tid=7558]]], state=connected] 2026-02-22T20:04:33.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=297, tag_tid=26, entry_tid=7561' 2026-02-22T20:04:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=297, tag_tid=26, entry_tid=7561' 2026-02-22T20:04:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=297, tag_tid=26, entry_tid=7561' '!=' '' 2026-02-22T20:04:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=297, tag_tid=26, entry_tid=7561' = '' 2026-02-22T20:04:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=297, tag_tid=26, entry_tid=7561' 2026-02-22T20:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:33.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=297, tag_tid=26, entry_tid=7561' 2026-02-22T20:04:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:33.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:33.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:33.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:33.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=297, tag_tid=26, entry_tid=7561' 2026-02-22T20:04:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:33.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=7561 2026-02-22T20:04:33.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 7561 2026-02-22T20:04:33.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:33.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:33.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:33.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:33.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:33.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:33.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:33.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:33.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:33.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:33.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:33.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:33.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:33.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:34.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:34.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:34.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:34.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:34.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:34.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:34.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:34.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:34.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:34.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:34.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:34.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:34.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:34.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:34.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:34.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:34.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:34.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:34.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:34.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:34.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:34.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:34.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:34.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:34.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:34.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:34.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=299, tag_tid=26, entry_tid=7891], [object_number=298, tag_tid=26, entry_tid=7890], [object_number=297, tag_tid=26, entry_tid=7889], [object_number=296, tag_tid=26, entry_tid=7888]]], state=connected] 2026-02-22T20:04:34.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:34.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:34.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:34.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:34.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:34.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:34.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=299, tag_tid=26, entry_tid=7891], [object_number=298, tag_tid=26, entry_tid=7890], [object_number=297, tag_tid=26, entry_tid=7889], [object_number=296, tag_tid=26, entry_tid=7888]]], state=connected] 2026-02-22T20:04:34.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=299, tag_tid=26, entry_tid=7891' 2026-02-22T20:04:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=299, tag_tid=26, entry_tid=7891' 2026-02-22T20:04:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=299, tag_tid=26, entry_tid=7891' '!=' 'object_number=297, tag_tid=26, entry_tid=7561' 2026-02-22T20:04:34.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:34.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=299, tag_tid=26, entry_tid=7891' = 'object_number=297, tag_tid=26, entry_tid=7561' 2026-02-22T20:04:34.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=299, tag_tid=26, entry_tid=7891' 2026-02-22T20:04:34.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:34.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:34.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:34.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=299, tag_tid=26, entry_tid=7891' 2026-02-22T20:04:34.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:34.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:34.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:34.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:34.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:34.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:34.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=299, tag_tid=26, entry_tid=7891' 2026-02-22T20:04:34.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:34.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:34.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=7891 2026-02-22T20:04:34.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 7891 2026-02-22T20:04:34.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:34.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:34.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:34.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:34.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:34.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:34.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:34.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:34.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:34.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:34.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:34.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:34.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:34.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:34.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:34.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:34.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:34.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:34.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:34.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:34.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:34.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:34.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:34.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:34.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:34.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:34.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:34.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:34.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:34.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:34.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:34.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:34.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:34.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:34.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:34.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:34.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:34.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:34.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:34.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:34.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:34.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:34.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:34.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:34.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:34.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:34.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:34.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:34.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:34.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:34.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:34.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:34.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:34.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:34.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:34.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:34.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:34.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:34.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:34.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:34.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:34.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:34.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:34.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:34.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:34.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=299, tag_tid=26, entry_tid=8227], [object_number=298, tag_tid=26, entry_tid=8226], [object_number=297, tag_tid=26, entry_tid=8225], [object_number=296, tag_tid=26, entry_tid=8224]]], state=connected] 2026-02-22T20:04:34.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:34.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:34.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:34.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=299, tag_tid=26, entry_tid=8227], [object_number=298, tag_tid=26, entry_tid=8226], [object_number=297, tag_tid=26, entry_tid=8225], [object_number=296, tag_tid=26, entry_tid=8224]]], state=connected] 2026-02-22T20:04:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=299, tag_tid=26, entry_tid=8227' 2026-02-22T20:04:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=299, tag_tid=26, entry_tid=8227' 2026-02-22T20:04:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=299, tag_tid=26, entry_tid=8227' '!=' 'object_number=299, tag_tid=26, entry_tid=7891' 2026-02-22T20:04:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:34.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=299, tag_tid=26, entry_tid=8227' = 'object_number=299, tag_tid=26, entry_tid=7891' 2026-02-22T20:04:34.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=299, tag_tid=26, entry_tid=8227' 2026-02-22T20:04:34.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:34.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:34.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:34.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:34.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=299, tag_tid=26, entry_tid=8227' 2026-02-22T20:04:34.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:34.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:34.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:34.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:34.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=299, tag_tid=26, entry_tid=8227' 2026-02-22T20:04:34.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:34.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:34.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=8227 2026-02-22T20:04:34.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 8227 2026-02-22T20:04:34.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:34.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:34.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:35.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:35.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:35.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:35.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:35.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:35.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:35.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:35.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:35.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:35.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:35.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:35.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:35.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:35.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:35.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:35.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:35.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:35.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:35.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:35.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:35.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:35.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:35.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:35.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:35.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:35.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:35.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:35.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:35.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:35.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:35.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:35.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:35.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:35.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:35.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:35.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:35.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:35.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:35.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:35.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:35.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:35.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=296, tag_tid=26, entry_tid=8508], [object_number=299, tag_tid=26, entry_tid=8507], [object_number=298, tag_tid=26, entry_tid=8506], [object_number=297, tag_tid=26, entry_tid=8505]]], state=connected] 2026-02-22T20:04:35.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:35.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:35.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:35.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:35.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:35.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:35.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:35.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:35.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:35.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:35.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=296, tag_tid=26, entry_tid=8508], [object_number=299, tag_tid=26, entry_tid=8507], [object_number=298, tag_tid=26, entry_tid=8506], [object_number=297, tag_tid=26, entry_tid=8505]]], state=connected] 2026-02-22T20:04:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=296, tag_tid=26, entry_tid=8508' 2026-02-22T20:04:35.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=296, tag_tid=26, entry_tid=8508' 2026-02-22T20:04:35.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=296, tag_tid=26, entry_tid=8508' '!=' 'object_number=299, tag_tid=26, entry_tid=8227' 2026-02-22T20:04:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=296, tag_tid=26, entry_tid=8508' = 'object_number=299, tag_tid=26, entry_tid=8227' 2026-02-22T20:04:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=296, tag_tid=26, entry_tid=8508' 2026-02-22T20:04:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:35.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:35.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:35.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=296, tag_tid=26, entry_tid=8508' 2026-02-22T20:04:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:35.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:35.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:35.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:35.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:35.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=296, tag_tid=26, entry_tid=8508' 2026-02-22T20:04:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:35.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=8508 2026-02-22T20:04:35.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 8508 2026-02-22T20:04:35.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:35.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:35.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:35.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:35.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:35.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:35.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:35.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:35.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:35.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:35.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:35.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:35.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:35.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:35.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:35.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:35.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:35.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:35.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:35.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:35.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:35.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:35.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:35.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:35.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:35.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:35.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:35.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:35.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:35.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:35.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:35.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:35.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:35.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:35.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:35.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:35.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:35.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:35.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:35.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:35.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:35.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:35.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:35.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:35.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:35.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=297, tag_tid=26, entry_tid=8833], [object_number=296, tag_tid=26, entry_tid=8832], [object_number=299, tag_tid=26, entry_tid=8831], [object_number=298, tag_tid=26, entry_tid=8830]]], state=connected] 2026-02-22T20:04:35.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:35.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:35.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:35.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:35.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:35.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:36.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:36.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:36.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:36.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:36.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=297, tag_tid=26, entry_tid=8833], [object_number=296, tag_tid=26, entry_tid=8832], [object_number=299, tag_tid=26, entry_tid=8831], [object_number=298, tag_tid=26, entry_tid=8830]]], state=connected] 2026-02-22T20:04:36.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:36.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=297, tag_tid=26, entry_tid=8833' 2026-02-22T20:04:36.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=297, tag_tid=26, entry_tid=8833' 2026-02-22T20:04:36.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=297, tag_tid=26, entry_tid=8833' '!=' 'object_number=296, tag_tid=26, entry_tid=8508' 2026-02-22T20:04:36.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:36.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=297, tag_tid=26, entry_tid=8833' = 'object_number=296, tag_tid=26, entry_tid=8508' 2026-02-22T20:04:36.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=297, tag_tid=26, entry_tid=8833' 2026-02-22T20:04:36.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:36.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:36.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:36.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=297, tag_tid=26, entry_tid=8833' 2026-02-22T20:04:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:36.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:36.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=297, tag_tid=26, entry_tid=8833' 2026-02-22T20:04:36.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:36.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:36.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=8833 2026-02-22T20:04:36.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 8833 2026-02-22T20:04:36.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:36.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:36.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:36.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:36.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:36.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:36.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:36.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:36.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:36.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:36.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:36.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:36.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:36.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:36.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:36.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:36.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:36.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:36.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:36.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:36.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:36.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:36.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:36.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:36.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:36.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:36.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:36.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:36.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:36.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:36.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:36.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:36.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:36.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:36.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:36.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:36.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:36.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:36.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:36.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:36.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:36.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:36.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:36.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=298, tag_tid=26, entry_tid=9178], [object_number=297, tag_tid=26, entry_tid=9177], [object_number=296, tag_tid=26, entry_tid=9176], [object_number=299, tag_tid=26, entry_tid=9175]]], state=connected] 2026-02-22T20:04:36.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:36.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:36.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:36.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:36.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:36.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:36.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:36.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:36.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:36.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=298, tag_tid=26, entry_tid=9178], [object_number=297, tag_tid=26, entry_tid=9177], [object_number=296, tag_tid=26, entry_tid=9176], [object_number=299, tag_tid=26, entry_tid=9175]]], state=connected] 2026-02-22T20:04:36.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:36.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=298, tag_tid=26, entry_tid=9178' 2026-02-22T20:04:36.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=298, tag_tid=26, entry_tid=9178' 2026-02-22T20:04:36.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=298, tag_tid=26, entry_tid=9178' '!=' 'object_number=297, tag_tid=26, entry_tid=8833' 2026-02-22T20:04:36.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:36.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=298, tag_tid=26, entry_tid=9178' = 'object_number=297, tag_tid=26, entry_tid=8833' 2026-02-22T20:04:36.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=298, tag_tid=26, entry_tid=9178' 2026-02-22T20:04:36.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:36.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:36.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:36.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=298, tag_tid=26, entry_tid=9178' 2026-02-22T20:04:36.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:36.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:36.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:36.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:36.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:36.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=298, tag_tid=26, entry_tid=9178' 2026-02-22T20:04:36.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:36.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:36.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=9178 2026-02-22T20:04:36.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 9178 2026-02-22T20:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:36.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:36.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:36.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:36.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:36.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:36.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:36.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:36.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:36.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:36.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:36.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:36.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:36.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:36.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:36.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:36.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:36.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:36.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:36.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:36.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:36.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:36.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:36.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:36.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:36.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:36.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:36.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:36.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:36.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:36.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:36.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:36.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:36.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:36.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:36.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:36.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:36.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:36.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:36.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:37.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:37.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:37.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:37.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:37.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:37.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:37.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:37.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:37.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:37.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:37.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:37.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:37.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:37.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:37.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:37.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:37.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:37.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=298, tag_tid=26, entry_tid=9490], [object_number=297, tag_tid=26, entry_tid=9489], [object_number=296, tag_tid=26, entry_tid=9488], [object_number=299, tag_tid=26, entry_tid=9487]]], state=connected] 2026-02-22T20:04:37.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:37.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:37.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:37.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:37.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:37.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:37.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=298, tag_tid=26, entry_tid=9490], [object_number=297, tag_tid=26, entry_tid=9489], [object_number=296, tag_tid=26, entry_tid=9488], [object_number=299, tag_tid=26, entry_tid=9487]]], state=connected] 2026-02-22T20:04:37.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=298, tag_tid=26, entry_tid=9490' 2026-02-22T20:04:37.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=298, tag_tid=26, entry_tid=9490' 2026-02-22T20:04:37.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=298, tag_tid=26, entry_tid=9490' '!=' 'object_number=298, tag_tid=26, entry_tid=9178' 2026-02-22T20:04:37.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:37.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=298, tag_tid=26, entry_tid=9490' = 'object_number=298, tag_tid=26, entry_tid=9178' 2026-02-22T20:04:37.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=298, tag_tid=26, entry_tid=9490' 2026-02-22T20:04:37.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:37.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:37.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:37.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:37.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=298, tag_tid=26, entry_tid=9490' 2026-02-22T20:04:37.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:37.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:37.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:37.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=298, tag_tid=26, entry_tid=9490' 2026-02-22T20:04:37.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:37.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=9490 2026-02-22T20:04:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 9490 2026-02-22T20:04:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:37.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:37.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:37.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:37.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:37.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:37.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:37.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:37.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:37.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:37.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:37.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:37.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:37.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:37.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:37.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:37.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:37.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:37.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:37.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:37.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:37.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:37.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:37.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:37.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:37.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:37.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:37.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:37.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:37.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:37.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:37.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:37.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:37.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=296, tag_tid=26, entry_tid=9808], [object_number=299, tag_tid=26, entry_tid=9807], [object_number=298, tag_tid=26, entry_tid=9806], [object_number=297, tag_tid=26, entry_tid=9805]]], state=connected] 2026-02-22T20:04:37.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:37.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:37.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:37.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:37.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 74 2026-02-22T20:04:37.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:37.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:37.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:37.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=296, tag_tid=26, entry_tid=9808], [object_number=299, tag_tid=26, entry_tid=9807], [object_number=298, tag_tid=26, entry_tid=9806], [object_number=297, tag_tid=26, entry_tid=9805]]], state=connected] 2026-02-22T20:04:37.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:37.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=296, tag_tid=26, entry_tid=9808' 2026-02-22T20:04:37.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=296, tag_tid=26, entry_tid=9808' 2026-02-22T20:04:37.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=296, tag_tid=26, entry_tid=9808' '!=' 'object_number=298, tag_tid=26, entry_tid=9490' 2026-02-22T20:04:37.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:37.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=296, tag_tid=26, entry_tid=9808' = 'object_number=298, tag_tid=26, entry_tid=9490' 2026-02-22T20:04:37.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=296, tag_tid=26, entry_tid=9808' 2026-02-22T20:04:37.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:37.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:37.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:37.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:37.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=296, tag_tid=26, entry_tid=9808' 2026-02-22T20:04:37.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:37.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:37.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:37.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:37.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:37.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:37.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:37.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=296, tag_tid=26, entry_tid=9808' 2026-02-22T20:04:37.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:37.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:37.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=9808 2026-02-22T20:04:37.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 9808 2026-02-22T20:04:37.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:37.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:37.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:37.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:37.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:37.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:37.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:37.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:37.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:37.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:37.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:37.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:37.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:37.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:37.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:37.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:37.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:37.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:37.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:37.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:37.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:37.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:37.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:37.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:38.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:38.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:38.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:38.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:38.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:38.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:38.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:38.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:38.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:38.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:38.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:38.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:38.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:38.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:38.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:38.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:38.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:38.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:38.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:38.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:38.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:38.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:38.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:38.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:38.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:38.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:38.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:38.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:38.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:38.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:38.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=300, tag_tid=26, entry_tid=10060], [object_number=303, tag_tid=26, entry_tid=10059], [object_number=302, tag_tid=26, entry_tid=10058], [object_number=301, tag_tid=26, entry_tid=10057]]], state=connected] 2026-02-22T20:04:38.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:38.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:38.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:38.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:38.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:38.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:38.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:38.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:38.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:38.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:38.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:38.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:38.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:38.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:38.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:38.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:38.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:38.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:38.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=300, tag_tid=26, entry_tid=10060], [object_number=303, tag_tid=26, entry_tid=10059], [object_number=302, tag_tid=26, entry_tid=10058], [object_number=301, tag_tid=26, entry_tid=10057]]], state=connected] 2026-02-22T20:04:38.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=300, tag_tid=26, entry_tid=10060' 2026-02-22T20:04:38.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=300, tag_tid=26, entry_tid=10060' 2026-02-22T20:04:38.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=300, tag_tid=26, entry_tid=10060' '!=' 'object_number=296, tag_tid=26, entry_tid=9808' 2026-02-22T20:04:38.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:38.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=300, tag_tid=26, entry_tid=10060' = 'object_number=296, tag_tid=26, entry_tid=9808' 2026-02-22T20:04:38.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=300, tag_tid=26, entry_tid=10060' 2026-02-22T20:04:38.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:38.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:38.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:38.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:38.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=300, tag_tid=26, entry_tid=10060' 2026-02-22T20:04:38.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:38.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:38.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:38.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:38.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:38.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:38.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:38.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=300, tag_tid=26, entry_tid=10060' 2026-02-22T20:04:38.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:38.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:38.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=10060 2026-02-22T20:04:38.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 10060 2026-02-22T20:04:38.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:38.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:38.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:38.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:38.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:38.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:38.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:38.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:38.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:38.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:38.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:38.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:38.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:38.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:38.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:38.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:38.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:38.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:38.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:38.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:38.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:38.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:38.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:38.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:38.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:38.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:38.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:38.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:38.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:38.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:38.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:38.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:38.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:38.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:38.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:38.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:38.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:38.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:38.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:38.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:38.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:38.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:38.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:38.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:38.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:38.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:38.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:38.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:38.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:38.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:38.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:38.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:38.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:38.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:38.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:38.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:38.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:38.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.745 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:04:38.745+0000 7f65e934d640 -1 reset not still connected to 0x55f6c7647450 2026-02-22T20:04:38.746 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:04:38.745+0000 7f65e934d640 -1 reset not still connected to 0x55f6c9629380 2026-02-22T20:04:38.746 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:04:38.745+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:04:38.746 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:04:38.745+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:04:38.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:38.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:38.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:38.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:38.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=10401], [object_number=300, tag_tid=26, entry_tid=10400], [object_number=303, tag_tid=26, entry_tid=10399], [object_number=302, tag_tid=26, entry_tid=10398]]], state=connected] 2026-02-22T20:04:38.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:38.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:38.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:38.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:38.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:38.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:38.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:38.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:38.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:38.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=10401], [object_number=300, tag_tid=26, entry_tid=10400], [object_number=303, tag_tid=26, entry_tid=10399], [object_number=302, tag_tid=26, entry_tid=10398]]], state=connected] 2026-02-22T20:04:38.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:38.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=301, tag_tid=26, entry_tid=10401' 2026-02-22T20:04:38.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=301, tag_tid=26, entry_tid=10401' 2026-02-22T20:04:38.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=10401' '!=' 'object_number=300, tag_tid=26, entry_tid=10060' 2026-02-22T20:04:38.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:38.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=10401' = 'object_number=300, tag_tid=26, entry_tid=10060' 2026-02-22T20:04:38.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=301, tag_tid=26, entry_tid=10401' 2026-02-22T20:04:38.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:38.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:38.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:38.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=10401' 2026-02-22T20:04:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:38.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:38.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=10401' 2026-02-22T20:04:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=10401 2026-02-22T20:04:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 10401 2026-02-22T20:04:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:39.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:39.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:39.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:39.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:39.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:39.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:39.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:39.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:39.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:39.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:39.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:39.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:39.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:39.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:39.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:39.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:39.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:39.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:39.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:39.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:39.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:39.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:39.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:39.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:39.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:39.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:39.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:39.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:39.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:39.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:39.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:39.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:39.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:39.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:39.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:39.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:39.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:39.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:39.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:39.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:39.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:39.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:39.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:39.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:39.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:39.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:39.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:39.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:39.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:39.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:39.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:39.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:39.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:39.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:39.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:39.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:39.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:39.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:39.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:39.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:39.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:39.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:39.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:39.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:39.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:39.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:39.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:39.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:39.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:39.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:39.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:39.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:39.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:39.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:39.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:39.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=302, tag_tid=26, entry_tid=10750], [object_number=301, tag_tid=26, entry_tid=10749], [object_number=300, tag_tid=26, entry_tid=10748], [object_number=303, tag_tid=26, entry_tid=10747]]], state=connected] 2026-02-22T20:04:39.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:39.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:39.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:39.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:39.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:39.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:39.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:39.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:39.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:39.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:39.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=302, tag_tid=26, entry_tid=10750], [object_number=301, tag_tid=26, entry_tid=10749], [object_number=300, tag_tid=26, entry_tid=10748], [object_number=303, tag_tid=26, entry_tid=10747]]], state=connected] 2026-02-22T20:04:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=302, tag_tid=26, entry_tid=10750' 2026-02-22T20:04:39.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=302, tag_tid=26, entry_tid=10750' 2026-02-22T20:04:39.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=302, tag_tid=26, entry_tid=10750' '!=' 'object_number=301, tag_tid=26, entry_tid=10401' 2026-02-22T20:04:39.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:39.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=302, tag_tid=26, entry_tid=10750' = 'object_number=301, tag_tid=26, entry_tid=10401' 2026-02-22T20:04:39.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=302, tag_tid=26, entry_tid=10750' 2026-02-22T20:04:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:39.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:39.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=302, tag_tid=26, entry_tid=10750' 2026-02-22T20:04:39.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:39.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:39.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:39.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:39.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:39.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=302, tag_tid=26, entry_tid=10750' 2026-02-22T20:04:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=10750 2026-02-22T20:04:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 10750 2026-02-22T20:04:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:39.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:39.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:39.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:39.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:39.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:39.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:39.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:39.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:39.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:39.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:39.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:39.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:39.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:39.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:39.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:39.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:39.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:39.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:39.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:39.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:39.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:39.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:39.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:39.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:39.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:39.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:39.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:39.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:39.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:39.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:39.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:39.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:39.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:39.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:39.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:39.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:39.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:39.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:39.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:39.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:39.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:39.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:39.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:39.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:39.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:39.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:39.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:39.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:39.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:39.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:39.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:39.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:39.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:39.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:39.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:39.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:39.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:39.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=11097], [object_number=300, tag_tid=26, entry_tid=11096], [object_number=303, tag_tid=26, entry_tid=11095], [object_number=302, tag_tid=26, entry_tid=11094]]], state=connected] 2026-02-22T20:04:39.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:39.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:39.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:39.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:39.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:39.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:39.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:39.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:39.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=11097], [object_number=300, tag_tid=26, entry_tid=11096], [object_number=303, tag_tid=26, entry_tid=11095], [object_number=302, tag_tid=26, entry_tid=11094]]], state=connected] 2026-02-22T20:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:39.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=301, tag_tid=26, entry_tid=11097' 2026-02-22T20:04:39.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=301, tag_tid=26, entry_tid=11097' 2026-02-22T20:04:39.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=11097' '!=' 'object_number=302, tag_tid=26, entry_tid=10750' 2026-02-22T20:04:39.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:39.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=11097' = 'object_number=302, tag_tid=26, entry_tid=10750' 2026-02-22T20:04:39.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=301, tag_tid=26, entry_tid=11097' 2026-02-22T20:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=11097' 2026-02-22T20:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:39.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:39.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:39.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:39.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=11097' 2026-02-22T20:04:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=11097 2026-02-22T20:04:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 11097 2026-02-22T20:04:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:40.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:40.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:40.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:40.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:40.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:40.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:40.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:40.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:40.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:40.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:40.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:40.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:40.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:40.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:40.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:40.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:40.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:40.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:40.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:40.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:40.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:40.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:40.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:40.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:40.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:40.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:40.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:40.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:40.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:40.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:40.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:40.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:40.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:40.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:40.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:40.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:40.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:40.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:40.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:40.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:40.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:40.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:40.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:40.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:40.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:40.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:40.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:40.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:40.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=11421], [object_number=300, tag_tid=26, entry_tid=11420], [object_number=303, tag_tid=26, entry_tid=11419], [object_number=302, tag_tid=26, entry_tid=11418]]], state=connected] 2026-02-22T20:04:40.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:40.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:40.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:40.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:40.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:40.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:40.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:40.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:40.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:40.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:40.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:40.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:40.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:40.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:40.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=11421], [object_number=300, tag_tid=26, entry_tid=11420], [object_number=303, tag_tid=26, entry_tid=11419], [object_number=302, tag_tid=26, entry_tid=11418]]], state=connected] 2026-02-22T20:04:40.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:40.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=301, tag_tid=26, entry_tid=11421' 2026-02-22T20:04:40.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=301, tag_tid=26, entry_tid=11421' 2026-02-22T20:04:40.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=11421' '!=' 'object_number=301, tag_tid=26, entry_tid=11097' 2026-02-22T20:04:40.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:40.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=11421' = 'object_number=301, tag_tid=26, entry_tid=11097' 2026-02-22T20:04:40.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=301, tag_tid=26, entry_tid=11421' 2026-02-22T20:04:40.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:40.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:40.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:40.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:40.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=11421' 2026-02-22T20:04:40.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:40.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:40.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=11421' 2026-02-22T20:04:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:40.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=11421 2026-02-22T20:04:40.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 11421 2026-02-22T20:04:40.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:40.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:40.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:40.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:40.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:40.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:40.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:40.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:40.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:40.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:40.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:40.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:40.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:40.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:40.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:40.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:40.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:40.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:40.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:40.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:40.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:40.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:40.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:40.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:40.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:40.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:40.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:40.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:40.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:40.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:40.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:40.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:40.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:40.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:40.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:40.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:41.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:41.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:41.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:41.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=11725], [object_number=300, tag_tid=26, entry_tid=11724], [object_number=303, tag_tid=26, entry_tid=11723], [object_number=302, tag_tid=26, entry_tid=11722]]], state=connected] 2026-02-22T20:04:41.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:41.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:41.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:41.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:41.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:41.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:41.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:41.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:41.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:41.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:41.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:41.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:41.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:41.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:41.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:41.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:41.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:41.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:41.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=11725], [object_number=300, tag_tid=26, entry_tid=11724], [object_number=303, tag_tid=26, entry_tid=11723], [object_number=302, tag_tid=26, entry_tid=11722]]], state=connected] 2026-02-22T20:04:41.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=301, tag_tid=26, entry_tid=11725' 2026-02-22T20:04:41.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=301, tag_tid=26, entry_tid=11725' 2026-02-22T20:04:41.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=11725' '!=' 'object_number=301, tag_tid=26, entry_tid=11421' 2026-02-22T20:04:41.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:41.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=11725' = 'object_number=301, tag_tid=26, entry_tid=11421' 2026-02-22T20:04:41.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=301, tag_tid=26, entry_tid=11725' 2026-02-22T20:04:41.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:41.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:41.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:41.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=11725' 2026-02-22T20:04:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:41.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:41.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=11725' 2026-02-22T20:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:41.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=11725 2026-02-22T20:04:41.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 11725 2026-02-22T20:04:41.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:41.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:41.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:41.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:41.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:41.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:41.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:41.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:41.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:41.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:41.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:41.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:41.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:41.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:41.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:41.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:41.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:41.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:41.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:41.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:41.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:41.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:41.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:41.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:41.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:41.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:41.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:41.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:41.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:41.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:41.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:41.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:41.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:41.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:41.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:41.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:41.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:41.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:41.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:41.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:41.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:41.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:41.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:41.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:41.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=303, tag_tid=26, entry_tid=12031], [object_number=302, tag_tid=26, entry_tid=12030], [object_number=301, tag_tid=26, entry_tid=12029], [object_number=300, tag_tid=26, entry_tid=12028]]], state=connected] 2026-02-22T20:04:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:41.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:41.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:41.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:41.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:41.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:41.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:41.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:41.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:41.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:41.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:41.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:41.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:41.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=303, tag_tid=26, entry_tid=12031], [object_number=302, tag_tid=26, entry_tid=12030], [object_number=301, tag_tid=26, entry_tid=12029], [object_number=300, tag_tid=26, entry_tid=12028]]], state=connected] 2026-02-22T20:04:41.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:41.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=303, tag_tid=26, entry_tid=12031' 2026-02-22T20:04:41.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=303, tag_tid=26, entry_tid=12031' 2026-02-22T20:04:41.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=303, tag_tid=26, entry_tid=12031' '!=' 'object_number=301, tag_tid=26, entry_tid=11725' 2026-02-22T20:04:41.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:41.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=303, tag_tid=26, entry_tid=12031' = 'object_number=301, tag_tid=26, entry_tid=11725' 2026-02-22T20:04:41.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=303, tag_tid=26, entry_tid=12031' 2026-02-22T20:04:41.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:41.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:41.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:41.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=303, tag_tid=26, entry_tid=12031' 2026-02-22T20:04:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:41.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:41.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:41.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:41.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:41.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:41.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=303, tag_tid=26, entry_tid=12031' 2026-02-22T20:04:41.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:41.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=12031 2026-02-22T20:04:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 12031 2026-02-22T20:04:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:41.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:41.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:41.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:41.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:41.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:41.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:41.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:41.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:41.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:41.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:41.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:41.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:41.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:41.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:41.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:41.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:41.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:41.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:41.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:41.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:41.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:41.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:41.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:41.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:41.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:41.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:41.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:41.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:41.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:41.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:41.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:41.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:41.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:41.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:41.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:41.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:41.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:41.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:42.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:42.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:42.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:42.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:42.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:42.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:42.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:42.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:42.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:42.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:42.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:42.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:42.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:42.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:42.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:42.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:42.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:42.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:42.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=302, tag_tid=26, entry_tid=12330], [object_number=301, tag_tid=26, entry_tid=12329], [object_number=300, tag_tid=26, entry_tid=12328], [object_number=303, tag_tid=26, entry_tid=12327]]], state=connected] 2026-02-22T20:04:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:42.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:42.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:42.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:42.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:42.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:42.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:42.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:42.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:42.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=302, tag_tid=26, entry_tid=12330], [object_number=301, tag_tid=26, entry_tid=12329], [object_number=300, tag_tid=26, entry_tid=12328], [object_number=303, tag_tid=26, entry_tid=12327]]], state=connected] 2026-02-22T20:04:42.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=302, tag_tid=26, entry_tid=12330' 2026-02-22T20:04:42.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=302, tag_tid=26, entry_tid=12330' 2026-02-22T20:04:42.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=302, tag_tid=26, entry_tid=12330' '!=' 'object_number=303, tag_tid=26, entry_tid=12031' 2026-02-22T20:04:42.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:42.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=302, tag_tid=26, entry_tid=12330' = 'object_number=303, tag_tid=26, entry_tid=12031' 2026-02-22T20:04:42.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=302, tag_tid=26, entry_tid=12330' 2026-02-22T20:04:42.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:42.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:42.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:42.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:42.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=302, tag_tid=26, entry_tid=12330' 2026-02-22T20:04:42.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:42.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:42.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:42.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=302, tag_tid=26, entry_tid=12330' 2026-02-22T20:04:42.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:42.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:42.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=12330 2026-02-22T20:04:42.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 12330 2026-02-22T20:04:42.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:42.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:42.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:42.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:42.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:42.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:42.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:42.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:42.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:42.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:42.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:42.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:42.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:42.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:42.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:42.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:42.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:42.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:42.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:42.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:42.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:42.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:42.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:42.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:42.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:42.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:42.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:42.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:42.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:42.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:42.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:42.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:42.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:42.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:42.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:42.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:42.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:42.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:42.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:42.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:42.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:42.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:42.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:42.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:42.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=300, tag_tid=26, entry_tid=12620], [object_number=303, tag_tid=26, entry_tid=12619], [object_number=302, tag_tid=26, entry_tid=12618], [object_number=301, tag_tid=26, entry_tid=12617]]], state=connected] 2026-02-22T20:04:42.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:42.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:42.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:42.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:42.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:42.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:42.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:42.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:42.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:42.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:42.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:42.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:42.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:42.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:42.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=300, tag_tid=26, entry_tid=12620], [object_number=303, tag_tid=26, entry_tid=12619], [object_number=302, tag_tid=26, entry_tid=12618], [object_number=301, tag_tid=26, entry_tid=12617]]], state=connected] 2026-02-22T20:04:42.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:42.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=300, tag_tid=26, entry_tid=12620' 2026-02-22T20:04:42.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=300, tag_tid=26, entry_tid=12620' 2026-02-22T20:04:42.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=300, tag_tid=26, entry_tid=12620' '!=' 'object_number=302, tag_tid=26, entry_tid=12330' 2026-02-22T20:04:42.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:42.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=300, tag_tid=26, entry_tid=12620' = 'object_number=302, tag_tid=26, entry_tid=12330' 2026-02-22T20:04:42.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=300, tag_tid=26, entry_tid=12620' 2026-02-22T20:04:42.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:42.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:42.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=300, tag_tid=26, entry_tid=12620' 2026-02-22T20:04:42.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:42.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:42.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:42.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:42.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:42.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=300, tag_tid=26, entry_tid=12620' 2026-02-22T20:04:42.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:42.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:42.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=12620 2026-02-22T20:04:42.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 12620 2026-02-22T20:04:42.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:42.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:42.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:42.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:42.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:42.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:42.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:42.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:42.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:42.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:42.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:42.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:42.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:42.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:43.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:43.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:43.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:43.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:43.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:43.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:43.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:43.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:43.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:43.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:43.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:43.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:43.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:43.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:43.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:43.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:43.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:43.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:43.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:43.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:43.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:43.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:43.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:43.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:43.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:43.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:43.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:43.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=303, tag_tid=26, entry_tid=12935], [object_number=302, tag_tid=26, entry_tid=12934], [object_number=301, tag_tid=26, entry_tid=12933], [object_number=300, tag_tid=26, entry_tid=12932]]], state=connected] 2026-02-22T20:04:43.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:43.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:43.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:43.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:43.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:43.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:43.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=303, tag_tid=26, entry_tid=12935], [object_number=302, tag_tid=26, entry_tid=12934], [object_number=301, tag_tid=26, entry_tid=12933], [object_number=300, tag_tid=26, entry_tid=12932]]], state=connected] 2026-02-22T20:04:43.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=303, tag_tid=26, entry_tid=12935' 2026-02-22T20:04:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=303, tag_tid=26, entry_tid=12935' 2026-02-22T20:04:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=303, tag_tid=26, entry_tid=12935' '!=' 'object_number=300, tag_tid=26, entry_tid=12620' 2026-02-22T20:04:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=303, tag_tid=26, entry_tid=12935' = 'object_number=300, tag_tid=26, entry_tid=12620' 2026-02-22T20:04:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=303, tag_tid=26, entry_tid=12935' 2026-02-22T20:04:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:43.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=303, tag_tid=26, entry_tid=12935' 2026-02-22T20:04:43.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:43.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:43.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=303, tag_tid=26, entry_tid=12935' 2026-02-22T20:04:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=12935 2026-02-22T20:04:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 12935 2026-02-22T20:04:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:43.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:43.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:43.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:43.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:43.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:43.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:43.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:43.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:43.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:43.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:43.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:43.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:43.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:43.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:43.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:43.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:43.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:43.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:43.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:43.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:43.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:43.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:43.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:43.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:43.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:43.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:43.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:43.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:43.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:43.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:43.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:43.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:43.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:43.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:43.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:43.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:43.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:43.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:43.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:43.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:43.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:43.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:43.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:43.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:43.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:43.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:43.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:43.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:43.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:43.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:43.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:43.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:43.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:43.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:43.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:43.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:43.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:43.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:43.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:43.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:43.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:43.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:43.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:43.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:43.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:43.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:43.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:43.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:43.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:43.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:43.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:43.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:43.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:43.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=302, tag_tid=26, entry_tid=13230], [object_number=301, tag_tid=26, entry_tid=13229], [object_number=300, tag_tid=26, entry_tid=13228], [object_number=303, tag_tid=26, entry_tid=13227]]], state=connected] 2026-02-22T20:04:43.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:43.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:43.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:43.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:43.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:43.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:43.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=302, tag_tid=26, entry_tid=13230], [object_number=301, tag_tid=26, entry_tid=13229], [object_number=300, tag_tid=26, entry_tid=13228], [object_number=303, tag_tid=26, entry_tid=13227]]], state=connected] 2026-02-22T20:04:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:43.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=302, tag_tid=26, entry_tid=13230' 2026-02-22T20:04:43.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=302, tag_tid=26, entry_tid=13230' 2026-02-22T20:04:43.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=302, tag_tid=26, entry_tid=13230' '!=' 'object_number=303, tag_tid=26, entry_tid=12935' 2026-02-22T20:04:43.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:43.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=302, tag_tid=26, entry_tid=13230' = 'object_number=303, tag_tid=26, entry_tid=12935' 2026-02-22T20:04:43.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=302, tag_tid=26, entry_tid=13230' 2026-02-22T20:04:43.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:43.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:43.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:43.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:43.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=302, tag_tid=26, entry_tid=13230' 2026-02-22T20:04:43.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:43.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:43.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:43.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:43.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:43.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=302, tag_tid=26, entry_tid=13230' 2026-02-22T20:04:43.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:43.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:43.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=13230 2026-02-22T20:04:43.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 13230 2026-02-22T20:04:43.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:43.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:43.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:44.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:44.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:44.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:44.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:44.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:44.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:44.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:44.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:44.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:44.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:44.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:44.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:44.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:44.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:44.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:44.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:44.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:44.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:44.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:44.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:44.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=302, tag_tid=26, entry_tid=13542], [object_number=301, tag_tid=26, entry_tid=13541], [object_number=300, tag_tid=26, entry_tid=13540], [object_number=303, tag_tid=26, entry_tid=13539]]], state=connected] 2026-02-22T20:04:44.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:44.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:44.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:44.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:44.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:44.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:44.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:44.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:44.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:44.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:44.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:44.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:44.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:44.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:44.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:44.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:44.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:44.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:44.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=302, tag_tid=26, entry_tid=13542], [object_number=301, tag_tid=26, entry_tid=13541], [object_number=300, tag_tid=26, entry_tid=13540], [object_number=303, tag_tid=26, entry_tid=13539]]], state=connected] 2026-02-22T20:04:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=302, tag_tid=26, entry_tid=13542' 2026-02-22T20:04:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=302, tag_tid=26, entry_tid=13542' 2026-02-22T20:04:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=302, tag_tid=26, entry_tid=13542' '!=' 'object_number=302, tag_tid=26, entry_tid=13230' 2026-02-22T20:04:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=302, tag_tid=26, entry_tid=13542' = 'object_number=302, tag_tid=26, entry_tid=13230' 2026-02-22T20:04:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=302, tag_tid=26, entry_tid=13542' 2026-02-22T20:04:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=302, tag_tid=26, entry_tid=13542' 2026-02-22T20:04:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:44.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:44.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:44.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:44.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:44.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:44.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=302, tag_tid=26, entry_tid=13542' 2026-02-22T20:04:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=13542 2026-02-22T20:04:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 13542 2026-02-22T20:04:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:44.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:44.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:44.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:44.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:44.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:44.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:44.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:44.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:44.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:44.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:44.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:44.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:44.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:44.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:44.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:44.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:44.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:44.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:44.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:44.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:44.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:44.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:44.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:44.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:44.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:44.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:44.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:44.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:44.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:44.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:44.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:44.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:44.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:44.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:44.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:44.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:44.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:44.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:44.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:44.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:44.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:44.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:44.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:44.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:44.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:44.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=13853], [object_number=300, tag_tid=26, entry_tid=13852], [object_number=303, tag_tid=26, entry_tid=13851], [object_number=302, tag_tid=26, entry_tid=13850]]], state=connected] 2026-02-22T20:04:44.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:44.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:44.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:44.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:44.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:44.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:44.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:44.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:44.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:44.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:44.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:44.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:44.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:44.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=301, tag_tid=26, entry_tid=13853], [object_number=300, tag_tid=26, entry_tid=13852], [object_number=303, tag_tid=26, entry_tid=13851], [object_number=302, tag_tid=26, entry_tid=13850]]], state=connected] 2026-02-22T20:04:45.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:45.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=301, tag_tid=26, entry_tid=13853' 2026-02-22T20:04:45.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=301, tag_tid=26, entry_tid=13853' 2026-02-22T20:04:45.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=13853' '!=' 'object_number=302, tag_tid=26, entry_tid=13542' 2026-02-22T20:04:45.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:45.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=301, tag_tid=26, entry_tid=13853' = 'object_number=302, tag_tid=26, entry_tid=13542' 2026-02-22T20:04:45.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=301, tag_tid=26, entry_tid=13853' 2026-02-22T20:04:45.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:45.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:45.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:45.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:45.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=13853' 2026-02-22T20:04:45.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:45.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:45.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=301, tag_tid=26, entry_tid=13853' 2026-02-22T20:04:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:45.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=13853 2026-02-22T20:04:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 13853 2026-02-22T20:04:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:45.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:45.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:45.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:45.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:45.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:45.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:45.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:45.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:45.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:45.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:45.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:45.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:45.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:45.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:45.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:45.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:45.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:45.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:45.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:45.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:45.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:45.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:45.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:45.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:45.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:45.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:45.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:45.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:45.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:45.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:45.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:45.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:45.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:45.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:45.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:45.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:45.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:45.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:45.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:45.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:45.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:45.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:45.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:45.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:45.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:45.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:45.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:45.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:45.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=300, tag_tid=26, entry_tid=14192], [object_number=303, tag_tid=26, entry_tid=14191], [object_number=302, tag_tid=26, entry_tid=14190], [object_number=301, tag_tid=26, entry_tid=14189]]], state=connected] 2026-02-22T20:04:45.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:45.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:45.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:45.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:45.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:45.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:45.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:45.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:45.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:45.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:45.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 75 2026-02-22T20:04:45.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:45.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:45.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:45.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=300, tag_tid=26, entry_tid=14192], [object_number=303, tag_tid=26, entry_tid=14191], [object_number=302, tag_tid=26, entry_tid=14190], [object_number=301, tag_tid=26, entry_tid=14189]]], state=connected] 2026-02-22T20:04:45.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:45.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=300, tag_tid=26, entry_tid=14192' 2026-02-22T20:04:45.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=300, tag_tid=26, entry_tid=14192' 2026-02-22T20:04:45.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=300, tag_tid=26, entry_tid=14192' '!=' 'object_number=301, tag_tid=26, entry_tid=13853' 2026-02-22T20:04:45.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:45.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=300, tag_tid=26, entry_tid=14192' = 'object_number=301, tag_tid=26, entry_tid=13853' 2026-02-22T20:04:45.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=300, tag_tid=26, entry_tid=14192' 2026-02-22T20:04:45.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:45.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:45.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:45.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:45.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=300, tag_tid=26, entry_tid=14192' 2026-02-22T20:04:45.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:45.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:45.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:45.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=300, tag_tid=26, entry_tid=14192' 2026-02-22T20:04:45.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:45.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:45.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=14192 2026-02-22T20:04:45.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 14192 2026-02-22T20:04:45.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:45.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:45.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:45.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:45.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:45.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:45.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:45.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:45.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:45.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:45.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:45.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:45.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:45.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:45.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:45.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:45.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:45.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:45.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:45.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:45.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:45.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:45.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:45.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:45.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:45.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:45.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:45.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:45.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:45.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:45.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:45.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:45.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:46.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:46.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:46.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:46.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:46.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:46.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:46.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:46.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:46.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:46.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:46.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:46.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:46.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:46.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:46.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:46.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:46.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:46.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:46.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:46.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=307, tag_tid=26, entry_tid=14443], [object_number=306, tag_tid=26, entry_tid=14442], [object_number=305, tag_tid=26, entry_tid=14441], [object_number=304, tag_tid=26, entry_tid=14440]]], state=connected] 2026-02-22T20:04:46.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:46.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:46.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:46.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:46.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:46.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:46.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:46.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:46.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:46.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=307, tag_tid=26, entry_tid=14443], [object_number=306, tag_tid=26, entry_tid=14442], [object_number=305, tag_tid=26, entry_tid=14441], [object_number=304, tag_tid=26, entry_tid=14440]]], state=connected] 2026-02-22T20:04:46.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=307, tag_tid=26, entry_tid=14443' 2026-02-22T20:04:46.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=307, tag_tid=26, entry_tid=14443' 2026-02-22T20:04:46.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=307, tag_tid=26, entry_tid=14443' '!=' 'object_number=300, tag_tid=26, entry_tid=14192' 2026-02-22T20:04:46.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:46.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=307, tag_tid=26, entry_tid=14443' = 'object_number=300, tag_tid=26, entry_tid=14192' 2026-02-22T20:04:46.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=307, tag_tid=26, entry_tid=14443' 2026-02-22T20:04:46.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:46.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:46.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:46.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:46.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=307, tag_tid=26, entry_tid=14443' 2026-02-22T20:04:46.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:46.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:46.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=307, tag_tid=26, entry_tid=14443' 2026-02-22T20:04:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:46.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=14443 2026-02-22T20:04:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 14443 2026-02-22T20:04:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:46.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:46.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:46.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:46.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:46.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:46.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:46.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:46.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:46.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:46.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:46.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:46.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:46.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:46.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:46.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:46.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:46.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:46.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:46.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:46.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:46.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:46.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:46.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:46.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:46.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:46.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:46.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:46.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:46.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:46.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:46.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:46.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:46.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:46.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:46.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:46.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:46.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:46.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:46.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:46.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:46.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:46.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:46.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:46.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:46.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:46.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:46.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:46.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:46.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:46.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:46.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:46.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:46.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:46.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:46.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:46.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:46.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:46.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:46.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:46.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:46.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=14740], [object_number=307, tag_tid=26, entry_tid=14739], [object_number=306, tag_tid=26, entry_tid=14738], [object_number=305, tag_tid=26, entry_tid=14737]]], state=connected] 2026-02-22T20:04:46.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:46.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:46.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:46.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:46.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=14740], [object_number=307, tag_tid=26, entry_tid=14739], [object_number=306, tag_tid=26, entry_tid=14738], [object_number=305, tag_tid=26, entry_tid=14737]]], state=connected] 2026-02-22T20:04:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=304, tag_tid=26, entry_tid=14740' 2026-02-22T20:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=304, tag_tid=26, entry_tid=14740' 2026-02-22T20:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=14740' '!=' 'object_number=307, tag_tid=26, entry_tid=14443' 2026-02-22T20:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=14740' = 'object_number=307, tag_tid=26, entry_tid=14443' 2026-02-22T20:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=304, tag_tid=26, entry_tid=14740' 2026-02-22T20:04:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:46.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:46.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=14740' 2026-02-22T20:04:46.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:46.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:46.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:46.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:46.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:46.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=14740' 2026-02-22T20:04:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:46.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:46.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=14740 2026-02-22T20:04:46.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 14740 2026-02-22T20:04:46.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:46.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:46.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:46.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:46.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:46.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:46.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:46.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:46.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:46.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:46.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:46.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:46.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:46.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:46.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:46.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:46.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:46.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:46.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:46.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:46.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:46.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:46.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:46.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:46.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:46.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:46.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:46.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:46.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:46.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:46.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:46.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:46.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:47.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:47.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:47.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:47.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:47.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:47.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:47.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:47.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:47.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:47.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:47.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:47.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:47.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:47.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:47.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:47.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:47.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:47.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:47.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:47.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:47.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:47.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:47.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:47.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:47.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:47.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:47.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:47.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:47.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:47.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:47.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:47.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:47.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=305, tag_tid=26, entry_tid=15093], [object_number=304, tag_tid=26, entry_tid=15092], [object_number=307, tag_tid=26, entry_tid=15091], [object_number=306, tag_tid=26, entry_tid=15090]]], state=connected] 2026-02-22T20:04:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:47.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:47.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:47.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:47.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:47.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:47.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:47.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:47.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:47.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:47.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:47.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:47.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:47.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:47.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=305, tag_tid=26, entry_tid=15093], [object_number=304, tag_tid=26, entry_tid=15092], [object_number=307, tag_tid=26, entry_tid=15091], [object_number=306, tag_tid=26, entry_tid=15090]]], state=connected] 2026-02-22T20:04:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=305, tag_tid=26, entry_tid=15093' 2026-02-22T20:04:47.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=305, tag_tid=26, entry_tid=15093' 2026-02-22T20:04:47.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=305, tag_tid=26, entry_tid=15093' '!=' 'object_number=304, tag_tid=26, entry_tid=14740' 2026-02-22T20:04:47.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:47.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=305, tag_tid=26, entry_tid=15093' = 'object_number=304, tag_tid=26, entry_tid=14740' 2026-02-22T20:04:47.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=305, tag_tid=26, entry_tid=15093' 2026-02-22T20:04:47.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:47.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:47.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:47.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:47.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=305, tag_tid=26, entry_tid=15093' 2026-02-22T20:04:47.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:47.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:47.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:47.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:47.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:47.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:47.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:47.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=305, tag_tid=26, entry_tid=15093' 2026-02-22T20:04:47.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:47.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:47.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=15093 2026-02-22T20:04:47.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 15093 2026-02-22T20:04:47.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:47.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:47.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:47.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:47.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:47.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:47.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:47.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:47.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:47.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:47.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:47.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:47.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:47.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:47.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:47.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:47.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:47.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:47.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:47.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:47.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:47.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:47.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:47.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:47.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:47.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:47.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:47.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:47.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:47.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:47.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:47.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:47.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:47.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:47.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:47.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:47.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:47.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:47.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:47.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:47.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:47.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:47.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:47.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:47.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:47.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:47.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:47.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:47.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:47.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:47.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:47.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:47.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:47.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:47.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:47.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:47.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:47.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:47.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:47.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:47.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:47.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:47.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:47.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:47.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:47.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:47.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:47.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:47.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:47.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:47.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:47.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:47.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:47.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:47.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=15432], [object_number=307, tag_tid=26, entry_tid=15431], [object_number=306, tag_tid=26, entry_tid=15430], [object_number=305, tag_tid=26, entry_tid=15429]]], state=connected] 2026-02-22T20:04:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:47.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:47.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=15432], [object_number=307, tag_tid=26, entry_tid=15431], [object_number=306, tag_tid=26, entry_tid=15430], [object_number=305, tag_tid=26, entry_tid=15429]]], state=connected] 2026-02-22T20:04:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:47.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=304, tag_tid=26, entry_tid=15432' 2026-02-22T20:04:47.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=304, tag_tid=26, entry_tid=15432' 2026-02-22T20:04:47.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=15432' '!=' 'object_number=305, tag_tid=26, entry_tid=15093' 2026-02-22T20:04:47.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:47.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=15432' = 'object_number=305, tag_tid=26, entry_tid=15093' 2026-02-22T20:04:47.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=304, tag_tid=26, entry_tid=15432' 2026-02-22T20:04:47.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:47.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:47.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:47.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:47.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=15432' 2026-02-22T20:04:47.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:47.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:47.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=15432' 2026-02-22T20:04:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=15432 2026-02-22T20:04:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 15432 2026-02-22T20:04:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:48.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:48.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:48.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:48.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:48.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:48.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:48.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:48.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:48.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:48.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:48.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:48.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:48.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:48.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:48.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:48.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:48.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:48.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:48.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:48.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:48.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:48.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:48.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:48.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:48.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:48.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:48.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:48.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:48.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:48.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:48.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:48.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:48.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:48.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:48.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:48.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:48.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:48.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:48.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:48.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:48.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:48.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:48.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:48.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:48.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:48.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:48.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:48.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:48.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:48.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:48.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:48.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:48.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:48.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:48.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:48.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:48.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=15744], [object_number=307, tag_tid=26, entry_tid=15743], [object_number=306, tag_tid=26, entry_tid=15742], [object_number=305, tag_tid=26, entry_tid=15741]]], state=connected] 2026-02-22T20:04:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:48.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:48.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:48.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:48.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:48.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:48.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:48.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:48.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=15744], [object_number=307, tag_tid=26, entry_tid=15743], [object_number=306, tag_tid=26, entry_tid=15742], [object_number=305, tag_tid=26, entry_tid=15741]]], state=connected] 2026-02-22T20:04:48.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=304, tag_tid=26, entry_tid=15744' 2026-02-22T20:04:48.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=304, tag_tid=26, entry_tid=15744' 2026-02-22T20:04:48.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=15744' '!=' 'object_number=304, tag_tid=26, entry_tid=15432' 2026-02-22T20:04:48.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:48.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=15744' = 'object_number=304, tag_tid=26, entry_tid=15432' 2026-02-22T20:04:48.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=304, tag_tid=26, entry_tid=15744' 2026-02-22T20:04:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:48.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:48.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:48.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=15744' 2026-02-22T20:04:48.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:48.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:48.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:48.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:48.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:48.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:48.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:48.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=15744' 2026-02-22T20:04:48.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:48.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=15744 2026-02-22T20:04:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 15744 2026-02-22T20:04:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:48.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:48.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:48.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:48.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:48.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:48.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:48.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:48.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:48.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:48.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:48.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:48.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:48.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:48.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:48.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:48.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:48.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:48.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:48.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:48.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:48.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:48.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:48.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:48.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:48.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:48.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:48.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:48.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:48.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:48.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:48.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:48.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:48.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:48.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:48.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:48.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:48.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:48.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:48.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:48.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:48.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:48.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:48.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:48.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:48.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:48.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:48.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:48.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:48.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=16076], [object_number=307, tag_tid=26, entry_tid=16075], [object_number=306, tag_tid=26, entry_tid=16074], [object_number=305, tag_tid=26, entry_tid=16073]]], state=connected] 2026-02-22T20:04:48.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=16076], [object_number=307, tag_tid=26, entry_tid=16075], [object_number=306, tag_tid=26, entry_tid=16074], [object_number=305, tag_tid=26, entry_tid=16073]]], state=connected] 2026-02-22T20:04:49.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:49.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=304, tag_tid=26, entry_tid=16076' 2026-02-22T20:04:49.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=304, tag_tid=26, entry_tid=16076' 2026-02-22T20:04:49.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=16076' '!=' 'object_number=304, tag_tid=26, entry_tid=15744' 2026-02-22T20:04:49.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:49.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=16076' = 'object_number=304, tag_tid=26, entry_tid=15744' 2026-02-22T20:04:49.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=304, tag_tid=26, entry_tid=16076' 2026-02-22T20:04:49.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:49.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:49.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:49.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:49.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=16076' 2026-02-22T20:04:49.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:49.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=16076' 2026-02-22T20:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=16076 2026-02-22T20:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 16076 2026-02-22T20:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:49.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:49.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:49.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:49.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:49.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:49.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:49.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:49.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:49.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:49.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:49.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:49.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:49.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:49.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:49.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:49.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:49.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:49.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:49.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:49.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:49.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:49.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:49.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:49.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:49.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:49.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=16392], [object_number=307, tag_tid=26, entry_tid=16391], [object_number=306, tag_tid=26, entry_tid=16390], [object_number=305, tag_tid=26, entry_tid=16389]]], state=connected] 2026-02-22T20:04:49.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:49.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:49.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:49.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:49.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:49.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:49.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:49.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:49.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:49.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:49.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:49.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=16392], [object_number=307, tag_tid=26, entry_tid=16391], [object_number=306, tag_tid=26, entry_tid=16390], [object_number=305, tag_tid=26, entry_tid=16389]]], state=connected] 2026-02-22T20:04:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:49.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=304, tag_tid=26, entry_tid=16392' 2026-02-22T20:04:49.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=304, tag_tid=26, entry_tid=16392' 2026-02-22T20:04:49.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=16392' '!=' 'object_number=304, tag_tid=26, entry_tid=16076' 2026-02-22T20:04:49.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:49.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=16392' = 'object_number=304, tag_tid=26, entry_tid=16076' 2026-02-22T20:04:49.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=304, tag_tid=26, entry_tid=16392' 2026-02-22T20:04:49.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:49.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=16392' 2026-02-22T20:04:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:49.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:49.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:49.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:49.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=16392' 2026-02-22T20:04:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:49.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=16392 2026-02-22T20:04:49.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 16392 2026-02-22T20:04:49.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:49.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:49.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:49.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:49.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:49.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:49.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:49.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:49.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:49.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:49.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:49.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:49.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:49.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:49.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:49.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:49.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:49.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:49.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:49.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:49.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:49.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:49.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:49.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:49.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:49.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:49.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:49.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:49.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:49.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:50.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:50.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:50.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:50.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:50.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:50.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=305, tag_tid=26, entry_tid=16721], [object_number=304, tag_tid=26, entry_tid=16720], [object_number=307, tag_tid=26, entry_tid=16719], [object_number=306, tag_tid=26, entry_tid=16718]]], state=connected] 2026-02-22T20:04:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:50.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:50.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:50.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:50.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:50.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:50.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:50.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=305, tag_tid=26, entry_tid=16721], [object_number=304, tag_tid=26, entry_tid=16720], [object_number=307, tag_tid=26, entry_tid=16719], [object_number=306, tag_tid=26, entry_tid=16718]]], state=connected] 2026-02-22T20:04:50.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=305, tag_tid=26, entry_tid=16721' 2026-02-22T20:04:50.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=305, tag_tid=26, entry_tid=16721' 2026-02-22T20:04:50.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=305, tag_tid=26, entry_tid=16721' '!=' 'object_number=304, tag_tid=26, entry_tid=16392' 2026-02-22T20:04:50.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:50.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=305, tag_tid=26, entry_tid=16721' = 'object_number=304, tag_tid=26, entry_tid=16392' 2026-02-22T20:04:50.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=305, tag_tid=26, entry_tid=16721' 2026-02-22T20:04:50.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:50.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:50.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:50.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:50.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=305, tag_tid=26, entry_tid=16721' 2026-02-22T20:04:50.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:50.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:50.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:50.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=305, tag_tid=26, entry_tid=16721' 2026-02-22T20:04:50.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:50.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=16721 2026-02-22T20:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 16721 2026-02-22T20:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:50.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:50.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:50.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:50.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:50.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:50.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:50.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:50.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:50.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:50.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:50.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:50.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:50.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:50.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:50.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:50.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:50.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:50.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:50.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:50.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:50.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:50.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:50.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:50.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:50.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:50.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:50.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:50.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:50.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:50.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:50.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:50.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:50.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:50.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:50.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:50.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:50.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:50.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:50.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:50.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=307, tag_tid=26, entry_tid=17047], [object_number=306, tag_tid=26, entry_tid=17046], [object_number=305, tag_tid=26, entry_tid=17045], [object_number=304, tag_tid=26, entry_tid=17044]]], state=connected] 2026-02-22T20:04:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:50.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:50.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:50.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:50.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:50.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:50.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=307, tag_tid=26, entry_tid=17047], [object_number=306, tag_tid=26, entry_tid=17046], [object_number=305, tag_tid=26, entry_tid=17045], [object_number=304, tag_tid=26, entry_tid=17044]]], state=connected] 2026-02-22T20:04:50.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:50.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=307, tag_tid=26, entry_tid=17047' 2026-02-22T20:04:50.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=307, tag_tid=26, entry_tid=17047' 2026-02-22T20:04:50.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=307, tag_tid=26, entry_tid=17047' '!=' 'object_number=305, tag_tid=26, entry_tid=16721' 2026-02-22T20:04:50.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:50.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=307, tag_tid=26, entry_tid=17047' = 'object_number=305, tag_tid=26, entry_tid=16721' 2026-02-22T20:04:50.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=307, tag_tid=26, entry_tid=17047' 2026-02-22T20:04:50.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:50.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:50.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:50.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:50.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=307, tag_tid=26, entry_tid=17047' 2026-02-22T20:04:50.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:50.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:50.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:50.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:50.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:50.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:50.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:50.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=307, tag_tid=26, entry_tid=17047' 2026-02-22T20:04:50.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:50.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:50.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=17047 2026-02-22T20:04:50.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 17047 2026-02-22T20:04:50.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:50.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:50.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:50.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:50.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:50.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:50.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:50.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:50.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:50.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:51.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:51.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:51.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:51.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:51.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:51.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:51.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:51.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:51.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=17348], [object_number=307, tag_tid=26, entry_tid=17347], [object_number=306, tag_tid=26, entry_tid=17346], [object_number=305, tag_tid=26, entry_tid=17345]]], state=connected] 2026-02-22T20:04:51.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:51.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:51.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:51.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:51.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:51.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:51.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:51.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:51.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=304, tag_tid=26, entry_tid=17348], [object_number=307, tag_tid=26, entry_tid=17347], [object_number=306, tag_tid=26, entry_tid=17346], [object_number=305, tag_tid=26, entry_tid=17345]]], state=connected] 2026-02-22T20:04:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=304, tag_tid=26, entry_tid=17348' 2026-02-22T20:04:51.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=304, tag_tid=26, entry_tid=17348' 2026-02-22T20:04:51.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=17348' '!=' 'object_number=307, tag_tid=26, entry_tid=17047' 2026-02-22T20:04:51.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:51.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=304, tag_tid=26, entry_tid=17348' = 'object_number=307, tag_tid=26, entry_tid=17047' 2026-02-22T20:04:51.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=304, tag_tid=26, entry_tid=17348' 2026-02-22T20:04:51.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:51.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:51.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:51.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=17348' 2026-02-22T20:04:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:51.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:51.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:51.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:51.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:51.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:51.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=304, tag_tid=26, entry_tid=17348' 2026-02-22T20:04:51.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:51.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=17348 2026-02-22T20:04:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 17348 2026-02-22T20:04:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:51.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:51.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:51.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:51.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:51.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:51.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:51.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:51.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:51.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:51.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:51.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:51.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:51.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:51.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:51.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:51.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:51.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:51.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:51.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:51.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:51.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:51.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:51.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:51.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:51.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:51.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:51.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:51.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:51.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:51.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:51.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:51.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:51.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:51.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:51.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:51.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:51.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:51.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:51.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:51.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:51.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:51.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=307, tag_tid=26, entry_tid=17659], [object_number=306, tag_tid=26, entry_tid=17658], [object_number=305, tag_tid=26, entry_tid=17657], [object_number=304, tag_tid=26, entry_tid=17656]]], state=connected] 2026-02-22T20:04:51.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:51.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:51.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:51.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:51.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:51.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:51.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:51.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:51.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:51.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:51.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:51.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=307, tag_tid=26, entry_tid=17659], [object_number=306, tag_tid=26, entry_tid=17658], [object_number=305, tag_tid=26, entry_tid=17657], [object_number=304, tag_tid=26, entry_tid=17656]]], state=connected] 2026-02-22T20:04:51.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=307, tag_tid=26, entry_tid=17659' 2026-02-22T20:04:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=307, tag_tid=26, entry_tid=17659' 2026-02-22T20:04:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=307, tag_tid=26, entry_tid=17659' '!=' 'object_number=304, tag_tid=26, entry_tid=17348' 2026-02-22T20:04:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=307, tag_tid=26, entry_tid=17659' = 'object_number=304, tag_tid=26, entry_tid=17348' 2026-02-22T20:04:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=307, tag_tid=26, entry_tid=17659' 2026-02-22T20:04:51.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:51.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:51.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:51.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=307, tag_tid=26, entry_tid=17659' 2026-02-22T20:04:51.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:51.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:51.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:51.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:51.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:51.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:51.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:51.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=307, tag_tid=26, entry_tid=17659' 2026-02-22T20:04:51.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:51.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:51.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=17659 2026-02-22T20:04:51.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 17659 2026-02-22T20:04:51.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:51.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:51.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:52.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:52.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:52.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:52.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:52.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:52.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:52.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:52.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:52.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:52.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:52.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:52.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:52.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:52.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:52.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:52.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:52.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:52.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:52.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:52.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:52.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:52.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:52.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:52.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:52.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:52.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:52.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:52.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:52.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:52.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:52.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:52.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:52.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:52.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:52.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:52.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:52.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:52.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:52.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:52.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:52.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=305, tag_tid=26, entry_tid=17997], [object_number=304, tag_tid=26, entry_tid=17996], [object_number=307, tag_tid=26, entry_tid=17995], [object_number=306, tag_tid=26, entry_tid=17994]]], state=connected] 2026-02-22T20:04:52.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:52.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:52.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:52.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:52.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:52.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:52.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:52.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:52.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:52.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:52.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=305, tag_tid=26, entry_tid=17997], [object_number=304, tag_tid=26, entry_tid=17996], [object_number=307, tag_tid=26, entry_tid=17995], [object_number=306, tag_tid=26, entry_tid=17994]]], state=connected] 2026-02-22T20:04:52.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:52.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=305, tag_tid=26, entry_tid=17997' 2026-02-22T20:04:52.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=305, tag_tid=26, entry_tid=17997' 2026-02-22T20:04:52.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=305, tag_tid=26, entry_tid=17997' '!=' 'object_number=307, tag_tid=26, entry_tid=17659' 2026-02-22T20:04:52.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:52.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=305, tag_tid=26, entry_tid=17997' = 'object_number=307, tag_tid=26, entry_tid=17659' 2026-02-22T20:04:52.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=305, tag_tid=26, entry_tid=17997' 2026-02-22T20:04:52.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:52.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:52.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:52.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:52.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=305, tag_tid=26, entry_tid=17997' 2026-02-22T20:04:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:52.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:52.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:52.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:52.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:52.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=305, tag_tid=26, entry_tid=17997' 2026-02-22T20:04:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=17997 2026-02-22T20:04:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 17997 2026-02-22T20:04:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:52.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:52.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:52.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:52.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:52.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:52.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:52.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:52.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:52.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:52.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:52.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:52.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:52.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:52.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:52.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:52.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:52.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:52.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:52.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:52.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:52.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:52.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:52.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:52.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:52.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:52.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:52.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:52.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:52.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:52.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=306, tag_tid=26, entry_tid=18326], [object_number=305, tag_tid=26, entry_tid=18325], [object_number=304, tag_tid=26, entry_tid=18324], [object_number=307, tag_tid=26, entry_tid=18323]]], state=connected] 2026-02-22T20:04:52.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:53.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:53.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:53.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:53.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:53.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:53.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:53.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:53.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:53.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 76 2026-02-22T20:04:53.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:53.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:53.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:53.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=306, tag_tid=26, entry_tid=18326], [object_number=305, tag_tid=26, entry_tid=18325], [object_number=304, tag_tid=26, entry_tid=18324], [object_number=307, tag_tid=26, entry_tid=18323]]], state=connected] 2026-02-22T20:04:53.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=306, tag_tid=26, entry_tid=18326' 2026-02-22T20:04:53.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=306, tag_tid=26, entry_tid=18326' 2026-02-22T20:04:53.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=306, tag_tid=26, entry_tid=18326' '!=' 'object_number=305, tag_tid=26, entry_tid=17997' 2026-02-22T20:04:53.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:53.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=306, tag_tid=26, entry_tid=18326' = 'object_number=305, tag_tid=26, entry_tid=17997' 2026-02-22T20:04:53.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=306, tag_tid=26, entry_tid=18326' 2026-02-22T20:04:53.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:53.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:53.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:53.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:53.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=306, tag_tid=26, entry_tid=18326' 2026-02-22T20:04:53.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:53.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:53.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:53.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:53.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:53.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:53.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:53.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=306, tag_tid=26, entry_tid=18326' 2026-02-22T20:04:53.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:53.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:53.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=18326 2026-02-22T20:04:53.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 18326 2026-02-22T20:04:53.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:53.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:53.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:53.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:53.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:53.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:53.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:53.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:53.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:53.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:53.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:53.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:53.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:53.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:53.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:53.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:53.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:53.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:53.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:53.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:53.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:53.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:53.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:53.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:53.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:53.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:53.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:53.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:53.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:53.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:53.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:53.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:53.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:53.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:53.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:53.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=18605], [object_number=308, tag_tid=26, entry_tid=18604], [object_number=311, tag_tid=26, entry_tid=18603], [object_number=310, tag_tid=26, entry_tid=18602]]], state=connected] 2026-02-22T20:04:53.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:53.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:53.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:53.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:53.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:53.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:53.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:53.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=18605], [object_number=308, tag_tid=26, entry_tid=18604], [object_number=311, tag_tid=26, entry_tid=18603], [object_number=310, tag_tid=26, entry_tid=18602]]], state=connected] 2026-02-22T20:04:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=309, tag_tid=26, entry_tid=18605' 2026-02-22T20:04:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=309, tag_tid=26, entry_tid=18605' 2026-02-22T20:04:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=309, tag_tid=26, entry_tid=18605' '!=' 'object_number=306, tag_tid=26, entry_tid=18326' 2026-02-22T20:04:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=309, tag_tid=26, entry_tid=18605' = 'object_number=306, tag_tid=26, entry_tid=18326' 2026-02-22T20:04:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=309, tag_tid=26, entry_tid=18605' 2026-02-22T20:04:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=26, entry_tid=18605' 2026-02-22T20:04:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:53.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:53.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=26, entry_tid=18605' 2026-02-22T20:04:53.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:53.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:53.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=18605 2026-02-22T20:04:53.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 18605 2026-02-22T20:04:53.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:53.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:53.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:53.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:53.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:53.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:53.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:53.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:53.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:53.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:53.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:53.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:53.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:53.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:53.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:53.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:53.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:53.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:53.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:53.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:53.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:53.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:53.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:53.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:53.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:53.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:53.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:53.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:54.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:54.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:54.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:54.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:54.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:54.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:54.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:54.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:54.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:54.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:54.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:54.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:54.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:54.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:54.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:54.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=310, tag_tid=26, entry_tid=18938], [object_number=309, tag_tid=26, entry_tid=18937], [object_number=308, tag_tid=26, entry_tid=18936], [object_number=311, tag_tid=26, entry_tid=18935]]], state=connected] 2026-02-22T20:04:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:54.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:54.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:54.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:54.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:54.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:54.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:54.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=310, tag_tid=26, entry_tid=18938], [object_number=309, tag_tid=26, entry_tid=18937], [object_number=308, tag_tid=26, entry_tid=18936], [object_number=311, tag_tid=26, entry_tid=18935]]], state=connected] 2026-02-22T20:04:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=310, tag_tid=26, entry_tid=18938' 2026-02-22T20:04:54.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=310, tag_tid=26, entry_tid=18938' 2026-02-22T20:04:54.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=310, tag_tid=26, entry_tid=18938' '!=' 'object_number=309, tag_tid=26, entry_tid=18605' 2026-02-22T20:04:54.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:54.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=310, tag_tid=26, entry_tid=18938' = 'object_number=309, tag_tid=26, entry_tid=18605' 2026-02-22T20:04:54.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=310, tag_tid=26, entry_tid=18938' 2026-02-22T20:04:54.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:54.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:54.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:54.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:54.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=310, tag_tid=26, entry_tid=18938' 2026-02-22T20:04:54.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:54.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:54.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:54.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:54.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:54.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:54.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:54.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=310, tag_tid=26, entry_tid=18938' 2026-02-22T20:04:54.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=18938 2026-02-22T20:04:54.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 18938 2026-02-22T20:04:54.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:54.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:54.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:54.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:54.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:54.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:54.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:54.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:54.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:54.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:54.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:54.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:54.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:54.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:54.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:54.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:54.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:54.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:54.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:54.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:54.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:54.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:54.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:54.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:54.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:54.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:54.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:54.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:54.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:54.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:54.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:54.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:54.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:54.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:54.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:54.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:54.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:54.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:54.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:54.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:54.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:54.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:54.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:54.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:54.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:54.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:54.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=310, tag_tid=26, entry_tid=19230], [object_number=309, tag_tid=26, entry_tid=19229], [object_number=308, tag_tid=26, entry_tid=19228], [object_number=311, tag_tid=26, entry_tid=19227]]], state=connected] 2026-02-22T20:04:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:54.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:54.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:54.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:54.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:54.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:54.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:54.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:54.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:54.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=310, tag_tid=26, entry_tid=19230], [object_number=309, tag_tid=26, entry_tid=19229], [object_number=308, tag_tid=26, entry_tid=19228], [object_number=311, tag_tid=26, entry_tid=19227]]], state=connected] 2026-02-22T20:04:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=310, tag_tid=26, entry_tid=19230' 2026-02-22T20:04:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=310, tag_tid=26, entry_tid=19230' 2026-02-22T20:04:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=310, tag_tid=26, entry_tid=19230' '!=' 'object_number=310, tag_tid=26, entry_tid=18938' 2026-02-22T20:04:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=310, tag_tid=26, entry_tid=19230' = 'object_number=310, tag_tid=26, entry_tid=18938' 2026-02-22T20:04:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=310, tag_tid=26, entry_tid=19230' 2026-02-22T20:04:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:54.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=310, tag_tid=26, entry_tid=19230' 2026-02-22T20:04:54.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:54.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:54.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:54.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:54.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:54.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:54.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:54.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=310, tag_tid=26, entry_tid=19230' 2026-02-22T20:04:54.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:54.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:54.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=19230 2026-02-22T20:04:54.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 19230 2026-02-22T20:04:54.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:54.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:54.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:54.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:54.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:54.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:54.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:54.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:54.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:54.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:54.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:54.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:54.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:54.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:54.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:54.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:54.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:54.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:54.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:54.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:54.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:54.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:54.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:54.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:54.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:54.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:54.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:54.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:54.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:54.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:54.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:54.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:54.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:54.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:55.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:55.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:55.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:55.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:55.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:55.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:55.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:55.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:55.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:55.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:55.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:55.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:55.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:55.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:55.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:55.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:55.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:55.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:55.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:55.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:55.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:55.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:55.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:55.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:55.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:55.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:55.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:55.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:55.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:55.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:55.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:55.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:55.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:55.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:55.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:55.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:55.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:55.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:55.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:55.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:55.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:55.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:55.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:55.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=309, tag_tid=26, entry_tid=19445' '!=' 'object_number=310, tag_tid=26, entry_tid=19230' 2026-02-22T20:04:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:04:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=309, tag_tid=26, entry_tid=19445' = 'object_number=310, tag_tid=26, entry_tid=19230' 2026-02-22T20:04:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:55.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:55.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:55.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:55.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=27 2026-02-22T20:04:55.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:55.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:04:55.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:55.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=26 2026-02-22T20:04:55.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:55.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:55.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:55.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:04:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:04:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:04:55.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=19445 2026-02-22T20:04:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 27 = 26 -a 1 -le 19445 2026-02-22T20:04:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:04:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:04:55.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:55.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:55.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:55.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:55.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:55.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:55.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:55.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:55.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:55.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:55.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:55.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:55.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:55.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:55.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:55.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:55.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:55.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:55.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:55.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:55.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:55.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:55.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:55.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:55.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:55.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:55.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:55.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:55.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:55.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:55.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:55.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:55.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:55.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:55.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:55.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:55.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:55.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:55.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:55.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:55.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:55.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:55.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:55.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:55.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:55.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:55.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:55.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:55.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:55.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:55.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:55.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=309, tag_tid=26, entry_tid=19445' '!=' 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:55.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:55.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:04:56.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:56.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:56.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:56.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:56.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:56.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:56.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:56.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:56.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:56.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:56.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:56.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:56.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:56.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:56.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:56.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:56.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:56.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:56.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:56.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:56.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:56.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:56.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:56.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:56.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:56.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:56.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:56.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:56.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:56.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:56.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:56.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:56.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:56.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:56.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:56.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:56.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:56.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:56.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:56.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:56.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:56.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:56.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:56.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:56.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:56.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:56.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:56.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:56.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:56.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:56.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:56.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:56.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:56.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:56.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:56.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:56.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:56.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:56.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:56.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:56.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:56.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:56.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:56.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:56.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:56.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:56.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:56.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:56.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:56.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:56.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:56.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:56.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:56.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:56.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:56.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:56.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:56.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:56.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:56.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:56.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:56.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:56.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:56.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:56.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:56.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:56.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:56.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=309, tag_tid=26, entry_tid=19445' '!=' 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:56.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:04:57.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:57.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:57.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:57.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:57.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:57.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:57.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:57.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:57.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:57.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:57.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:57.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:57.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:57.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:57.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:57.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:57.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:57.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:57.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:57.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:57.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:57.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:57.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:57.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:57.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:57.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:57.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:57.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:57.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:57.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:57.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:57.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:57.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:57.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:57.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:57.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:57.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:57.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:57.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:57.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:57.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:57.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:57.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:57.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:57.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:57.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:57.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:57.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:57.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:57.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:57.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:57.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:57.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:57.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:57.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:57.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:57.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:57.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:57.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:57.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:57.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:57.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:57.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:57.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:57.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:57.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:57.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:57.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:57.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:57.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:57.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:57.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:57.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:57.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:57.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:57.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:57.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:57.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:57.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:57.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:57.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:57.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:57.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:57.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:57.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:57.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:57.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:57.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:57.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:57.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:57.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:57.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:57.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:57.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:57.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:57.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=309, tag_tid=26, entry_tid=19445' '!=' 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:04:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:04:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:04:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:04:59.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:59.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:04:59.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:59.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:04:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:04:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:04:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:59.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:59.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:59.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:59.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:04:59.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:59.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:59.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:59.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:59.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:04:59.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:04:59.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:59.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:59.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:04:59.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:04:59.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:04:59.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:04:59.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:59.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:59.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:04:59.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:04:59.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:59.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:59.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:59.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:59.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:04:59.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:04:59.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:59.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:59.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:04:59.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:04:59.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:59.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:04:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:04:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:04:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:04:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:04:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:04:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:04:59.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:59.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:04:59.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:04:59.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:04:59.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:04:59.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:04:59.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:04:59.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:04:59.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:04:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:04:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:04:59.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:59.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:59.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:04:59.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:04:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:04:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:04:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:04:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:04:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:59.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:59.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:59.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:59.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:59.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:59.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:59.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:04:59.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:04:59.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:59.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:59.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:59.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:04:59.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:04:59.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:04:59.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:04:59.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:04:59.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:59.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:04:59.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:59.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:59.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:04:59.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:59.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:04:59.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:04:59.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:04:59.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:59.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=26, entry_tid=19445], [object_number=308, tag_tid=26, entry_tid=19444], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:04:59.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:04:59.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:59.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:59.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=309, tag_tid=26, entry_tid=19445' '!=' 'object_number=309, tag_tid=26, entry_tid=19445' 2026-02-22T20:04:59.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:59.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:05:01.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:05:01.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:01.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:01.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:01.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:05:01.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:05:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:05:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:05:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:05:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:01.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:01.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:01.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:01.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:05:02.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:02.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:02.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:02.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:02.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:05:02.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:05:02.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:02.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:02.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:02.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:02.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:02.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:02.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:02.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:02.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:02.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:05:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:05:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:05:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:02.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:02.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:02.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:05:02.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:02.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:02.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:05:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:05:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:02.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:02.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:02.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:02.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:02.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:02.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:05:02.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:02.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:02.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:05:02.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:05:02.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:02.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:05:02.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:05:02.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:02.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:05:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:05:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:05:02.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:02.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:02.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:02.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:05:02.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:02.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:02.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:02.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 77 2026-02-22T20:05:02.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 77 2026-02-22T20:05:02.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:02.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:05:02.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=309, tag_tid=27, entry_tid=1], [object_number=308, tag_tid=27, entry_tid=0], [object_number=311, tag_tid=26, entry_tid=19443], [object_number=310, tag_tid=26, entry_tid=19442]]], state=connected] 2026-02-22T20:05:02.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=309, tag_tid=27, entry_tid=1' -a 'object_number=309, tag_tid=27, entry_tid=1' = 'object_number=309, tag_tid=27, entry_tid=1' 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap9 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:05:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:05:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2026-02-22T20:05:03.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T20:05:03.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:05:03.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:05:03.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 17 2026-02-22T20:05:03.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:03:36 2026 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:04:46 2026 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:05:03.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:05:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:05:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:05:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n snap8-clean ']' 2026-02-22T20:05:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap8-clean 2026-02-22T20:05:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap8-clean 2026-02-22T20:05:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:05:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:05:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:05:03.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:05:03.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:04.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:05:04.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:04.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap9 2026-02-22T20:05:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap9 2026-02-22T20:05:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:05:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:05:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:04.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:04.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:05:05.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:05:05.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:05.683 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:05:05.680+0000 7f65d9eff640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2026-02-22T20:05:05.683 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:05:05.680+0000 7f65d56f6640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2026-02-22T20:05:05.683 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:05:05.680+0000 7f65d8efd640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2026-02-22T20:05:05.684 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:05:05.680+0000 7f65d96fe640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2026-02-22T20:05:05.684 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:05:05.680+0000 7f65d6ef9640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2026-02-22T20:05:05.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:05:05.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:06.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:06.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:06.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap9-clean 2026-02-22T20:05:06.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap9-clean 2026-02-22T20:05:06.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:05:06.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:06.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:06.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap9-clean 2026-02-22T20:05:06.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T20:05:06.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:05:06.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror test 2026-02-22T20:05:06.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:05:06.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:06.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:05:06.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=6 2026-02-22T20:05:06.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:05:06.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-22T20:05:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 66491392 bytes 2026-02-22T20:05:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 102320128 bytes 2026-02-22T20:05:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 106907648 bytes 2026-02-22T20:05:09.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 115092480 bytes 2026-02-22T20:05:10.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 204295168 bytes 2026-02-22T20:05:12.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 311904256 bytes 2026-02-22T20:05:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:05:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:05:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:05:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:05:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:12.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:12.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:12.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:05:12.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:05:12.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:12.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:05:13.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:05:13.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:13.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:13.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:13.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:13.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local description_pattern= 2026-02-22T20:05:13.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local service_pattern= 2026-02-22T20:05:13.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:05:13.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:05:13.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:05:13.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='test: 2026-02-22T20:05:13.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: global_id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:05:13.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+replaying 2026-02-22T20:05:13.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: replaying, {"bytes_per_second":6576290.5,"entries_behind_primary":0,"entries_per_second":417.87,"non_primary_position":{"entry_tid":1,"object_number":309,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":309,"tag_tid":27}} 2026-02-22T20:05:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: service: mirror.3 on trial066 2026-02-22T20:05:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:05:03 2026-02-22T20:05:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: peer_sites: 2026-02-22T20:05:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: name: cluster2 2026-02-22T20:05:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: state: up+stopped 2026-02-22T20:05:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: description: local image is primary 2026-02-22T20:05:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: last_update: 2026-02-22 20:05:03' 2026-02-22T20:05:13.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:05:13.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: state: up+replaying 2026-02-22T20:05:13.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^ description: .*' 2026-02-22T20:05:13.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: description: replaying, {"bytes_per_second":6576290.5,"entries_behind_primary":0,"entries_per_second":417.87,"non_primary_position":{"entry_tid":1,"object_number":309,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":309,"tag_tid":27}} 2026-02-22T20:05:13.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:05:13.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:05:13.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep '^up+' 2026-02-22T20:05:13.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:05:13.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:13.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: service: mirror.3 on trial066 2026-02-22T20:05:13.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update 2026-02-22T20:05:13.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:13.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:05:03' 2026-02-22T20:05:13.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:05:03' 2026-02-22T20:05:13.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:13.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:13.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:13.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:13.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'prev_last_update=2026-02-22 20:05:03' 2026-02-22T20:05:13.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status 2026-02-22T20:05:13.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ CEPH_ARGS= 2026-02-22T20:05:13.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:13.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status='OKOKOK142134266mirror.3trial06619.2.3-689-gf90f4eaetrueOK43154318mirror.1trial06619.2.3-689-gf90f4eaefalseOK43894391mirror.2trial06619.2.3-689-gf90f4eaefalseOK44624465mirror.0trial06619.2.3-689-gf90f4eaefalseOKtest93a727b7-96e1-4ad4-b5af-d03cdbcc3c2eup+replayingreplaying, {"bytes_per_second":6576290.5,"entries_behind_primary":0,"entries_per_second":417.87,"non_primary_position":{"entry_tid":1,"object_number":309,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":309,"tag_tid":27}}42134266mirror.3trial0662026-02-22 20:05:03cluster29d155999-fdff-4f3a-a022-c51b5908dfe6up+stoppedlocal image is primary2026-02-22 20:05:03' 2026-02-22T20:05:13.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_update state 2026-02-22T20:05:13.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:13.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_update='2026-02-22 20:05:03' 2026-02-22T20:05:13.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:13.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=up+replaying 2026-02-22T20:05:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo up+replaying 2026-02-22T20:05:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep up+replaying 2026-02-22T20:05:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:up+replaying 2026-02-22T20:05:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap10 2026-02-22T20:05:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap10 2026-02-22T20:05:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:05:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap10 2026-02-22T20:05:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:05:19.646 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-22T20:05:19.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-22T20:05:19.646 INFO:tasks.rbd_mirror_thrash:waiting for 2.0501502562266607 secs before reviving daemons 2026-02-22T20:05:19.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:05:19.632+0000 7f1f57168640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 56437) UID: 1000 2026-02-22T20:05:21.697 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-22T20:05:21.698 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:05:21.698 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-22T20:05:21.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-22T20:05:21.698 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-22T20:05:21.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-22T20:05:21.701 INFO:tasks.rbd_mirror_thrash:waiting for 62.39718758962556 secs before thrashing 2026-02-22T20:05:26.725 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.47279473078541356) > weight (0.25) 2026-02-22T20:05:26.726 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-22T20:05:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-22T20:05:26.726 INFO:tasks.rbd_mirror_thrash:waiting for 4.125455956515736 secs before reviving daemons 2026-02-22T20:05:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:05:26.712+0000 7fc58c612640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 34846) UID: 1000 2026-02-22T20:05:30.852 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-22T20:05:30.852 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:05:30.853 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-22T20:05:30.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-22T20:05:30.853 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-22T20:05:30.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-22T20:05:30.856 INFO:tasks.rbd_mirror_thrash:waiting for 115.36337018044014 secs before thrashing 2026-02-22T20:05:44.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:05:44.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:44.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:44.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:44.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:05:44.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:44.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:44.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:44.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:05:44.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:05:44.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:44.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:05:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:05:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:45.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:45.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:45.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:05:45.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:05:45.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:05:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:05:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:05:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:05:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:05:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:05:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:05:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:05:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:05:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:05:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:05:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:05:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:45.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:45.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:05:45.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:05:45.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:45.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:45.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:45.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:45.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:05:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T20:05:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:45.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:45.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:45.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:45.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:05:45.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:05:45.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:45.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:45.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:45.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:45.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T20:05:45.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:45.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:05:45.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:45.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:45.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T20:05:45.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:05:45.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:05:45.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:45.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:45.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:05:45.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:05:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:45.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:45.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:45.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:45.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T20:05:45.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:45.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:05:45.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:45.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:45.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:45.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:45.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:45.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:05:45.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:05:45.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T20:05:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:05:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:05:45.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:05:45.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:05:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:05:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:05:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:05:46.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:05:46.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:46.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:46.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:46.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:05:46.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:46.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:05:46.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:05:46.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:46.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:46.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:46.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:46.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:46.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:46.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:46.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:05:46.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:46.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:46.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:05:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:05:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:46.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:46.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:46.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:46.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:05:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:46.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:05:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:05:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:05:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:05:46.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:46.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:46.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:46.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:46.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:05:46.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:05:46.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:46.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:46.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:46.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:46.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:46.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:46.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:46.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:46.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:46.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:46.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:46.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:46.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:05:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:46.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:46.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=324, tag_tid=29, entry_tid=19048], [object_number=327, tag_tid=29, entry_tid=19047], [object_number=326, tag_tid=29, entry_tid=19046], [object_number=325, tag_tid=29, entry_tid=19045]]], state=connected] 2026-02-22T20:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:05:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:46.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:46.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:46.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:46.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:46.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:46.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:46.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=324, tag_tid=29, entry_tid=19048], [object_number=327, tag_tid=29, entry_tid=19047], [object_number=326, tag_tid=29, entry_tid=19046], [object_number=325, tag_tid=29, entry_tid=19045]]], state=connected] 2026-02-22T20:05:46.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=324, tag_tid=29, entry_tid=19048' 2026-02-22T20:05:46.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=30, entry_tid=1' -a 'object_number=325, tag_tid=30, entry_tid=1' = 'object_number=324, tag_tid=29, entry_tid=19048' 2026-02-22T20:05:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=324, tag_tid=29, entry_tid=19048' '!=' '' 2026-02-22T20:05:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:05:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=324, tag_tid=29, entry_tid=19048' = '' 2026-02-22T20:05:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=324, tag_tid=29, entry_tid=19048' 2026-02-22T20:05:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:46.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:05:46.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:46.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=30 2026-02-22T20:05:46.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=324, tag_tid=29, entry_tid=19048' 2026-02-22T20:05:46.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:05:46.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:46.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=29 2026-02-22T20:05:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:05:46.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:46.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:05:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=324, tag_tid=29, entry_tid=19048' 2026-02-22T20:05:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:05:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=19048 2026-02-22T20:05:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 30 = 29 -a 1 -le 19048 2026-02-22T20:05:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:05:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:05:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:05:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:05:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:05:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:05:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:05:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:05:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:05:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:05:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:05:46.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:46.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:05:46.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:46.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:05:46.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:05:46.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:46.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:46.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:46.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:46.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:05:46.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:46.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:46.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:05:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:05:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:46.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:46.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:46.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:46.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:46.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:46.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:05:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:46.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:46.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:46.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:46.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:05:46.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:46.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:05:46.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:05:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:05:46.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:46.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:46.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:46.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:46.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:46.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:46.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:46.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=29, entry_tid=19373], [object_number=324, tag_tid=29, entry_tid=19372], [object_number=327, tag_tid=29, entry_tid=19371], [object_number=326, tag_tid=29, entry_tid=19370]]], state=connected] 2026-02-22T20:05:46.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:46.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:46.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:46.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:05:46.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:46.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:46.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:46.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:46.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:46.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:47.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:47.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:47.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:47.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:47.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=29, entry_tid=19373], [object_number=324, tag_tid=29, entry_tid=19372], [object_number=327, tag_tid=29, entry_tid=19371], [object_number=326, tag_tid=29, entry_tid=19370]]], state=connected] 2026-02-22T20:05:47.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:47.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=325, tag_tid=29, entry_tid=19373' 2026-02-22T20:05:47.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=30, entry_tid=1' -a 'object_number=325, tag_tid=30, entry_tid=1' = 'object_number=325, tag_tid=29, entry_tid=19373' 2026-02-22T20:05:47.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=325, tag_tid=29, entry_tid=19373' '!=' 'object_number=324, tag_tid=29, entry_tid=19048' 2026-02-22T20:05:47.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:05:47.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=325, tag_tid=29, entry_tid=19373' = 'object_number=324, tag_tid=29, entry_tid=19048' 2026-02-22T20:05:47.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=325, tag_tid=29, entry_tid=19373' 2026-02-22T20:05:47.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:47.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:05:47.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:47.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=30 2026-02-22T20:05:47.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=29, entry_tid=19373' 2026-02-22T20:05:47.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:05:47.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:47.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=29 2026-02-22T20:05:47.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:47.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:05:47.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:47.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:05:47.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=29, entry_tid=19373' 2026-02-22T20:05:47.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:05:47.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:47.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=19373 2026-02-22T20:05:47.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 30 = 29 -a 1 -le 19373 2026-02-22T20:05:47.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:05:47.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:47.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:05:47.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:05:47.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:05:47.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:47.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:05:47.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:47.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:05:47.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:47.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:47.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:05:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:05:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:05:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:47.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:47.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:47.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:47.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:05:47.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:47.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:47.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:47.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:05:47.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:47.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:47.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:47.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:47.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:05:47.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:47.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:47.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:47.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:47.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:47.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:05:47.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:05:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:47.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:47.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:47.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:47.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:47.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:47.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:47.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:47.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:47.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:47.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:47.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:47.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:05:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:47.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:47.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:47.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:47.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:47.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:47.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:47.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:47.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:47.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:47.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:47.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:47.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:47.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=30, entry_tid=1' -a 'object_number=325, tag_tid=30, entry_tid=1' = 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:47.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=326, tag_tid=29, entry_tid=19622' '!=' 'object_number=325, tag_tid=29, entry_tid=19373' 2026-02-22T20:05:47.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:05:47.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=326, tag_tid=29, entry_tid=19622' = 'object_number=325, tag_tid=29, entry_tid=19373' 2026-02-22T20:05:47.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:47.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:47.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:05:47.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:47.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=30 2026-02-22T20:05:47.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:47.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:05:47.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:47.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=29 2026-02-22T20:05:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:05:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:47.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:05:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:05:47.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:05:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=19622 2026-02-22T20:05:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 30 = 29 -a 1 -le 19622 2026-02-22T20:05:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:05:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:47.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:47.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:05:47.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:47.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:05:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:05:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:05:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:47.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:05:47.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:47.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:47.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:05:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:05:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:47.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:05:47.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:47.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:47.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:47.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:47.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:05:47.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:47.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:47.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:47.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:47.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:05:47.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:05:47.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:47.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:47.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:47.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:47.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:47.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:47.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:47.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:47.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:05:48.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:48.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:48.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:05:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:05:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:05:48.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:48.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:48.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:48.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:48.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:05:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:05:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:05:48.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:48.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:48.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:48.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:05:48.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:48.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:48.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:48.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:48.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:48.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:48.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:48.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:48.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:48.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:05:48.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:48.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:48.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:48.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:48.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:48.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:48.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=30, entry_tid=1' -a 'object_number=325, tag_tid=30, entry_tid=1' = 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=326, tag_tid=29, entry_tid=19622' '!=' 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:05:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:05:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:05:48.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:48.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:48.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:48.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:48.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:48.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:05:48.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:48.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:48.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:48.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:48.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:48.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:48.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:48.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:48.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:05:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:05:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:05:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:05:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:48.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:48.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:48.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:48.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:05:48.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:48.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:48.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:48.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:48.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:48.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:48.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:48.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:48.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:48.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:48.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:48.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:48.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:05:48.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:48.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:48.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:48.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:48.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:48.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:48.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:48.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:48.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:05:48.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:48.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:48.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:48.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:05:48.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:48.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:48.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:48.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:05:48.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:48.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:48.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:48.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:48.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=30, entry_tid=1' -a 'object_number=325, tag_tid=30, entry_tid=1' = 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=326, tag_tid=29, entry_tid=19622' '!=' 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:05:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:05:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:05:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:49.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:05:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:05:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:49.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:49.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:49.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:49.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:49.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:49.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:49.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:49.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:49.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:49.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:49.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:49.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:05:49.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:49.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:49.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:49.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:05:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:49.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:49.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:49.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:49.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:49.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:49.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:49.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:49.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:49.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:05:49.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:49.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:49.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:49.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:49.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:05:49.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:05:49.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:49.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:49.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:49.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:49.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:49.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:49.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:49.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:05:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:49.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:49.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:49.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:49.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:05:50.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:50.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:05:50.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:50.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:50.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:50.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:05:50.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:50.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:50.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:50.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:05:50.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:50.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:50.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:50.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:50.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:50.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:50.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:50.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:50.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:05:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:05:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:05:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:50.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=30, entry_tid=1' -a 'object_number=325, tag_tid=30, entry_tid=1' = 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=326, tag_tid=29, entry_tid=19622' '!=' 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:05:51.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:05:51.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:51.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:51.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:51.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:05:51.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:05:51.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:05:51.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:05:51.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:05:51.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:51.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:51.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:05:51.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:05:51.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:51.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:05:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:05:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:05:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:05:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:51.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:51.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:51.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:05:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:51.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:51.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:05:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:05:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:51.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:51.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:51.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:51.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:05:51.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:51.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:51.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:51.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:51.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:05:51.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:51.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:51.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:05:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:51.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:51.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:51.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:51.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:05:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:05:51.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:52.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:52.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:52.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:52.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:52.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:52.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:52.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:05:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:52.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:52.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:52.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=326, tag_tid=29, entry_tid=19622], [object_number=325, tag_tid=29, entry_tid=19621], [object_number=324, tag_tid=29, entry_tid=19620], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:52.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:52.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=30, entry_tid=1' -a 'object_number=325, tag_tid=30, entry_tid=1' = 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=326, tag_tid=29, entry_tid=19622' '!=' 'object_number=326, tag_tid=29, entry_tid=19622' 2026-02-22T20:05:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:05:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:05:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:54.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:05:54.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:54.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:05:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:05:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:05:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:05:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:05:54.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:54.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:54.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:54.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:54.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:05:54.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:05:54.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:54.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:54.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:05:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:54.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:05:54.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:05:54.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:05:54.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:05:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:05:54.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:54.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:54.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:54.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:54.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:05:54.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:05:54.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:54.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:05:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:05:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:05:54.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:54.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:54.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:05:54.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:05:54.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:05:54.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:05:54.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:05:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:05:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:05:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:05:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:05:54.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:05:54.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:05:54.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:54.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:54.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:05:54.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:05:54.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:05:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:54.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:05:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:05:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:54.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:54.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:54.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:54.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:54.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:54.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:54.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:54.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:54.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:54.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:54.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:54.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:05:54.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:54.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:54.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:05:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:05:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:05:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:05:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:05:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:05:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:05:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:05:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:05:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:05:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:05:54.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:05:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=30, entry_tid=1' -a 'object_number=325, tag_tid=30, entry_tid=1' = 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:05:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2026-02-22T20:05:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:54.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap10 2026-02-22T20:05:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:05:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:05:54.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:05:55.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2026-02-22T20:05:55.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T20:05:55.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:05:55.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:05:55.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 19 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:05:03 2026 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:05:47 2026 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:05:55.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T20:05:55.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:05:55.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:05:55.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:05:55.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:05:55.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:55.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n snap9-clean ']' 2026-02-22T20:05:55.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap9-clean 2026-02-22T20:05:55.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:55.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:55.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap9-clean 2026-02-22T20:05:55.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:05:55.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:05:55.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:55.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:55.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:05:55.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:05:55.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:56.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:05:56.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:56.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:56.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:56.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap10 2026-02-22T20:05:56.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:56.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:56.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap10 2026-02-22T20:05:56.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:05:56.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:05:56.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:56.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:56.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:05:57.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:05:57.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:58.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:05:58.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:58.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:05:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:05:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ clean_snap_name=snap10-clean 2026-02-22T20:05:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror test snap10-clean 2026-02-22T20:05:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:05:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap10-clean 2026-02-22T20:05:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2026-02-22T20:05:59.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:05:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:05:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:05:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:05:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:05:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:05:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:05:59.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:59.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:06:00.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:06:00.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:00.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:00.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:00.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:06:00.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:06:00.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:06:00.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:06:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:06:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:06:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:06:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:06:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:06:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:06:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:06:00.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:06:00.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:00.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:00.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:00.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:00.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:06:00.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:00.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:00.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:00.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:06:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:06:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:06:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:06:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T20:06:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T20:06:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:06:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:06:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:06:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:06:00.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:06:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:00.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:00.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:00.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:00.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T20:06:00.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:00.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:00.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:00.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:00.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T20:06:00.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:06:00.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:06:00.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:00.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:06:00.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:06:00.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:06:00.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:06:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:06:00.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:00.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:00.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:00.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T20:06:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:06:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:06:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:06:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:06:00.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:06:00.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:06:00.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:00.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:00.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:06:00.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:06:00.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:06:00.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:00.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:06:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:06:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:06:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:06:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:06:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:06:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:06:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:06:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:06:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:06:01.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:01.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:06:01.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:01.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:06:01.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:06:01.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:01.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:01.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:01.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:01.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:06:01.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:01.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:01.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:01.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:01.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:06:01.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:06:01.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:01.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:06:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:06:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:06:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:06:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:06:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:01.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:01.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:01.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:01.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:06:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:01.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:06:01.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:06:01.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:06:01.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:01.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:06:01.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:06:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:06:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:01.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:01.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:06:01.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:01.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:01.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:01.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:01.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:01.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:06:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:06:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:06:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:01.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:01.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:01.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:01.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=31, entry_tid=1' -a 'object_number=325, tag_tid=31, entry_tid=1' = 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:01.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=325, tag_tid=30, entry_tid=1' '!=' '' 2026-02-22T20:06:01.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:06:01.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=325, tag_tid=30, entry_tid=1' = '' 2026-02-22T20:06:01.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:01.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:01.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:06:01.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:06:01.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=31 2026-02-22T20:06:01.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:01.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:06:01.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:06:01.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=30 2026-02-22T20:06:01.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:01.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:06:01.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:06:01.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:06:01.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:01.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:06:01.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:06:01.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=1 2026-02-22T20:06:01.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 31 = 30 -a 1 -le 1 2026-02-22T20:06:01.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:06:01.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:01.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:06:01.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:06:01.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:01.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:01.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:01.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:06:01.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:06:01.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:06:01.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:06:01.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:06:01.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:01.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:06:01.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:06:01.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:01.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:06:01.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:01.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:06:01.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:01.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:06:01.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:06:01.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:01.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:01.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:06:01.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:06:01.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:01.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:01.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:01.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:01.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:06:01.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:01.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:01.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:06:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:06:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:01.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:01.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:01.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:01.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:06:01.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:01.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:01.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:06:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:01.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:01.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:01.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:01.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:06:01.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:01.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:01.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:01.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:01.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:06:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:06:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:06:01.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:01.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:01.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:01.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:06:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:06:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:01.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:01.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:01.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:02.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:02.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:02.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:02.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:02.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:02.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:06:02.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:02.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:02.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:02.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:06:02.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:02.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:02.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:02.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:06:02.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:02.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:02.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:02.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:02.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:02.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:02.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=31, entry_tid=1' -a 'object_number=325, tag_tid=31, entry_tid=1' = 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=325, tag_tid=30, entry_tid=1' '!=' 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:02.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:06:02.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:06:02.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:02.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:02.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:06:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:06:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:06:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:06:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:06:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:06:02.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:06:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:06:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:02.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:02.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:02.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:02.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:06:02.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:02.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:02.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:02.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:02.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:06:02.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:06:02.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:02.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:02.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:02.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:06:02.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:06:02.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:02.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:02.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:02.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:02.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:06:02.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:02.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:02.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:02.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:02.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:02.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:02.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:02.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:02.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:06:02.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:06:02.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:02.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:02.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:02.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:02.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:06:02.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:02.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:02.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:06:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:06:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:06:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:06:02.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:02.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:02.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:02.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:06:02.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:02.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:06:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:06:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:06:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:02.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:02.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=31, entry_tid=1' -a 'object_number=325, tag_tid=31, entry_tid=1' = 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=325, tag_tid=30, entry_tid=1' '!=' 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:06:03.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:06:03.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:03.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:03.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:03.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:06:03.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:06:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:03.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:06:03.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:03.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:06:03.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:06:03.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:03.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:03.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:03.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:06:03.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:06:03.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:03.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:03.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:03.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:03.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:06:03.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:03.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:03.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:06:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:06:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:06:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:03.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:06:03.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:03.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:03.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:03.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:03.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:06:03.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:03.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:03.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:03.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:03.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:06:03.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:03.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:03.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:03.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:03.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:06:03.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:06:03.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:03.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:03.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:03.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:03.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:06:03.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:03.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:03.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:06:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:06:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:06:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:06:03.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:06:03.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:03.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:03.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:03.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:06:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:03.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:06:03.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:03.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:03.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:03.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:06:03.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:03.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:03.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:03.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:06:03.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:03.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:03.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:03.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:03.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:03.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:03.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:03.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:03.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:03.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:03.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:03.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:06:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:06:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:06:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:04.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:04.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:04.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:04.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:04.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:04.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=31, entry_tid=1' -a 'object_number=325, tag_tid=31, entry_tid=1' = 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=325, tag_tid=30, entry_tid=1' '!=' 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:04.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:06:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:06:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:05.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:06:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:06:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:06:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:06:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:06:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:06:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:06:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:05.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:05.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:05.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:05.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:05.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:05.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:06:05.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:06:05.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:05.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:06:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:06:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:06:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:06:05.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:06:05.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:05.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:05.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:05.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:05.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:06:05.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:05.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:05.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:05.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:05.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:06:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:06:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:06:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:05.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:05.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:05.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:06:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:05.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:05.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:05.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:05.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:05.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:05.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:05.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:05.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:06:05.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:05.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:05.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:05.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:06:05.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:05.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:05.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:05.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:06:05.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:05.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:05.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=31, entry_tid=1' -a 'object_number=325, tag_tid=31, entry_tid=1' = 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=325, tag_tid=30, entry_tid=1' '!=' 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:06:07.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:06:07.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:06:07.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:07.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:06:07.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:06:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:06:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:06:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:06:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:07.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:07.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:07.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:07.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:06:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:06:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:06:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:06:08.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:06:08.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:08.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:08.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:08.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:08.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:06:08.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:08.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:08.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:08.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:08.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:06:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:06:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:08.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:06:08.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:08.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:08.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:06:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:06:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:06:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:06:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:06:08.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:06:08.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:08.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:08.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:08.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:06:08.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:08.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:08.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:08.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:08.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:06:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:06:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:06:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:08.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:08.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:08.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:08.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:08.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:08.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:08.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:08.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:08.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:08.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=30, entry_tid=1], [object_number=324, tag_tid=30, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:08.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:08.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:08.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=31, entry_tid=1' -a 'object_number=325, tag_tid=31, entry_tid=1' = 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:08.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=325, tag_tid=30, entry_tid=1' '!=' 'object_number=325, tag_tid=30, entry_tid=1' 2026-02-22T20:06:08.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:08.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:06:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror test 2026-02-22T20:06:10.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:10.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:10.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:10.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:06:10.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n test ']' 2026-02-22T20:06:10.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-22T20:06:10.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-22T20:06:10.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:06:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:06:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:06:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:06:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:06:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-22T20:06:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:06:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:10.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:10.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:10.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:06:10.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:06:10.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:10.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:10.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:10.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:10.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-22T20:06:10.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:10.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:10.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:10.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:10.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2026-02-22T20:06:10.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:06:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:06:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:06:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2026-02-22T20:06:10.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:10.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:10.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:10.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:10.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:10.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:06:10.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:10.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:10.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:10.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:06:10.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:06:10.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:10.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:10.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:10.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2026-02-22T20:06:10.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T20:06:10.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:06:10.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:06:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2026-02-22T20:06:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:06:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:06:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:06:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:06:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:06:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:06:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:06:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:06:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:06:10.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:10.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:10.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:10.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2026-02-22T20:06:10.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:10.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-22T20:06:10.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:10.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:10.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:10.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-22T20:06:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:06:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:10.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:10.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:10.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:10.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:10.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:10.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:10.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:10.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:10.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:10.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-22T20:06:10.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-test.status 2026-02-22T20:06:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:10.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:10.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-22T20:06:10.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 81 2026-02-22T20:06:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 81 2026-02-22T20:06:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:06:10.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:10.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=325, tag_tid=31, entry_tid=1], [object_number=324, tag_tid=31, entry_tid=0], [object_number=326, tag_tid=29, entry_tid=19622], [object_number=327, tag_tid=29, entry_tid=19619]]], state=connected] 2026-02-22T20:06:10.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.status 2026-02-22T20:06:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=325, tag_tid=31, entry_tid=1' -a 'object_number=325, tag_tid=31, entry_tid=1' = 'object_number=325, tag_tid=31, entry_tid=1' 2026-02-22T20:06:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:10.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2026-02-22T20:06:10.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:06:10.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:10.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:10.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap10-clean 2026-02-22T20:06:10.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:06:10.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:06:10.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:06:11.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2026-02-22T20:06:11.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'test': 2026-02-22T20:06:11.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:06:11.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 20 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10a1190b0dfb 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10a1190b0dfb 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 19:53:35 2026 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:05:03 2026 2026-02-22T20:06:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:05:47 2026 2026-02-22T20:06:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:06:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10a1190b0dfb 2026-02-22T20:06:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:06:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:06:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 93a727b7-96e1-4ad4-b5af-d03cdbcc3c2e 2026-02-22T20:06:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:06:11.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:11.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 10 2026-02-22T20:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap1 2026-02-22T20:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap1 2026-02-22T20:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap1 2026-02-22T20:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:12.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:12.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:12.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:13.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap1-clean 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap1-clean 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap1-clean 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:13.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:14.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:14.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:14.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:14.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:14.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap2 2026-02-22T20:06:14.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap2 2026-02-22T20:06:14.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:14.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:14.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap2 2026-02-22T20:06:14.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:14.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:14.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:14.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:14.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:15.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:15.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:15.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:15.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:16.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:16.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap2-clean 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap2-clean 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap2-clean 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:16.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:16.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:17.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:17.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap3 2026-02-22T20:06:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap3 2026-02-22T20:06:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap3 2026-02-22T20:06:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:18.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:18.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:18.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:18.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:18.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap3-clean 2026-02-22T20:06:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap3-clean 2026-02-22T20:06:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap3-clean 2026-02-22T20:06:18.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:18.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:18.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:18.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:18.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:19.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:19.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:19.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:20.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:20.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:20.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:20.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:20.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap4 2026-02-22T20:06:20.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap4 2026-02-22T20:06:20.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:20.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:20.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap4 2026-02-22T20:06:20.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:20.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:20.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:20.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:20.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:20.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:20.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:21.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:21.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:21.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap4-clean 2026-02-22T20:06:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap4-clean 2026-02-22T20:06:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap4-clean 2026-02-22T20:06:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:22.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:22.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:22.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:22.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:23.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap5 2026-02-22T20:06:23.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap5 2026-02-22T20:06:23.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:23.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:23.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap5 2026-02-22T20:06:23.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:23.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:23.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:23.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:23.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:23.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:23.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:24.099 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-22T20:06:24.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-22T20:06:24.099 INFO:tasks.rbd_mirror_thrash:waiting for 5.48598831380097 secs before reviving daemons 2026-02-22T20:06:24.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:06:24.088+0000 7fe37fa7b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 80676) UID: 1000 2026-02-22T20:06:24.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:24.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:24.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:24.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap5-clean 2026-02-22T20:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap5-clean 2026-02-22T20:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap5-clean 2026-02-22T20:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:24.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:25.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:25.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:25.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:26.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap6 2026-02-22T20:06:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap6 2026-02-22T20:06:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap6 2026-02-22T20:06:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:26.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:27.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:27.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:27.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:27.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:27.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap6-clean 2026-02-22T20:06:27.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap6-clean 2026-02-22T20:06:27.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:27.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:27.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap6-clean 2026-02-22T20:06:27.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:27.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:27.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:27.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:27.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:28.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:28.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:28.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:29.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:29.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap7 2026-02-22T20:06:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap7 2026-02-22T20:06:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap7 2026-02-22T20:06:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:29.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:29.586 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-22T20:06:29.586 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:06:29.587 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-22T20:06:29.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-22T20:06:29.587 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-22T20:06:29.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-22T20:06:29.588 INFO:tasks.rbd_mirror_thrash:waiting for 60.13015042934795 secs before thrashing 2026-02-22T20:06:30.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:30.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:30.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:30.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap7-clean 2026-02-22T20:06:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap7-clean 2026-02-22T20:06:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap7-clean 2026-02-22T20:06:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:30.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:30.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:31.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:31.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:31.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:31.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:31.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap8 2026-02-22T20:06:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap8 2026-02-22T20:06:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap8 2026-02-22T20:06:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:32.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:33.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:33.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:33.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:33.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap8-clean 2026-02-22T20:06:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap8-clean 2026-02-22T20:06:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap8-clean 2026-02-22T20:06:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:33.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:33.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:33.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:34.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:34.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:34.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:34.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:34.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap9 2026-02-22T20:06:34.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap9 2026-02-22T20:06:34.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:34.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:34.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap9 2026-02-22T20:06:34.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:34.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:34.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:34.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:34.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:35.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:35.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:36.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:36.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:36.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:36.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap9-clean 2026-02-22T20:06:36.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap9-clean 2026-02-22T20:06:36.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:36.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap9-clean 2026-02-22T20:06:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:36.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:36.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:37.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:37.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap10 2026-02-22T20:06:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap10 2026-02-22T20:06:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap10 2026-02-22T20:06:37.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:37.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:37.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:37.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:37.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:38.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:38.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:38.751 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:06:38.749+0000 7f65d8efd640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2026-02-22T20:06:38.752 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:06:38.749+0000 7f65d9eff640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2026-02-22T20:06:38.752 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:06:38.749+0000 7f65d56f6640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2026-02-22T20:06:38.752 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:06:38.749+0000 7f65d66f8640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2026-02-22T20:06:38.752 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:06:38.749+0000 7f65d96fe640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2026-02-22T20:06:38.752 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:06:38.749+0000 7f65daf01640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2026-02-22T20:06:38.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:38.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:39.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:39.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:39.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap10-clean 2026-02-22T20:06:39.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror test snap10-clean 2026-02-22T20:06:39.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap10-clean 2026-02-22T20:06:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:06:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export 2026-02-22T20:06:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:39.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:40.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:40.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-test.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-test.export 2026-02-22T20:06:40.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:06:40.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 10 2026-02-22T20:06:40.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:40.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap1 2026-02-22T20:06:40.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap1 2026-02-22T20:06:40.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:40.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:40.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:40.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap1 2026-02-22T20:06:40.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2026-02-22T20:06:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:41.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap1-clean 2026-02-22T20:06:41.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2026-02-22T20:06:41.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:41.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:41.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:41.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap1-clean 2026-02-22T20:06:41.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2026-02-22T20:06:42.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:42.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap2 2026-02-22T20:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap2 2026-02-22T20:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap2 2026-02-22T20:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2026-02-22T20:06:43.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:43.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap2-clean 2026-02-22T20:06:43.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2026-02-22T20:06:43.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:43.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:43.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:43.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap2-clean 2026-02-22T20:06:43.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2026-02-22T20:06:44.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:44.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap3 2026-02-22T20:06:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap3 2026-02-22T20:06:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap3 2026-02-22T20:06:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2026-02-22T20:06:45.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap3-clean 2026-02-22T20:06:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2026-02-22T20:06:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap3-clean 2026-02-22T20:06:45.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2026-02-22T20:06:46.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap4 2026-02-22T20:06:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap4 2026-02-22T20:06:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap4 2026-02-22T20:06:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2026-02-22T20:06:47.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap4-clean 2026-02-22T20:06:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2026-02-22T20:06:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap4-clean 2026-02-22T20:06:47.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2026-02-22T20:06:48.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:48.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:48.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap5 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap5 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap5 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2026-02-22T20:06:49.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap5-clean 2026-02-22T20:06:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2026-02-22T20:06:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap5-clean 2026-02-22T20:06:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2026-02-22T20:06:50.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:50.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:50.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap6 2026-02-22T20:06:50.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap6 2026-02-22T20:06:50.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:50.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:50.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:50.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap6 2026-02-22T20:06:50.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2026-02-22T20:06:51.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap6-clean 2026-02-22T20:06:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2026-02-22T20:06:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap6-clean 2026-02-22T20:06:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2026-02-22T20:06:52.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:52.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:52.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap7 2026-02-22T20:06:52.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap7 2026-02-22T20:06:52.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:52.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:52.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:52.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap7 2026-02-22T20:06:52.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2026-02-22T20:06:53.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:53.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap7-clean 2026-02-22T20:06:53.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2026-02-22T20:06:53.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:53.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:53.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:53.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap7-clean 2026-02-22T20:06:53.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2026-02-22T20:06:54.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap8 2026-02-22T20:06:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap8 2026-02-22T20:06:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap8 2026-02-22T20:06:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2026-02-22T20:06:55.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap8-clean 2026-02-22T20:06:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2026-02-22T20:06:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap8-clean 2026-02-22T20:06:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2026-02-22T20:06:56.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:56.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:56.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap9 2026-02-22T20:06:56.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap9 2026-02-22T20:06:56.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:56.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:56.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:56.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap9 2026-02-22T20:06:56.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2026-02-22T20:06:57.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap9-clean 2026-02-22T20:06:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2026-02-22T20:06:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap9-clean 2026-02-22T20:06:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2026-02-22T20:06:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:06:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap10 2026-02-22T20:06:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap10 2026-02-22T20:06:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap10 2026-02-22T20:06:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2026-02-22T20:06:59.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap10-clean 2026-02-22T20:06:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2026-02-22T20:06:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:06:59.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:06:59.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:06:59.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap10-clean 2026-02-22T20:06:59.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2026-02-22T20:07:00.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:07:00.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror test 2026-02-22T20:07:00.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:00.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:00.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:07:00.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:07:00.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:07:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror test 2026-02-22T20:07:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:07:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2026-02-22T20:07:00.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/test 2026-02-22T20:07:01.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:07:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror test deleted 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:07:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:07:01.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:07:01.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:07:01.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:07:01.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:07:01.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:07:01.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:07:01.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:07:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror test deleted '' 2026-02-22T20:07:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:07:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=test 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=test 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:07:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:01.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:07:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:07:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:07:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:07:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:07:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ testlog 'TEST: create many images' 2026-02-22T20:07:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ tee -a /tmp/tmp.Xb7Kt1LMAh/rbd-mirror.test.log 2026-02-22T20:07:01.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ date '+%F %T' 2026-02-22T20:07:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo 2026-02-22 20:07:01 TEST: create many images 2026-02-22T20:07:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:2026-02-22 20:07:01 TEST: create many images 2026-02-22T20:07:01.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ snap_name=snap 2026-02-22T20:07:01.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 50 2026-02-22T20:07:01.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_1 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 journal 128M 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:07:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_1 128M 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_1 2026-02-22T20:07:01.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:01.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_1 2026-02-22T20:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:07:01.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=11 2026-02-22T20:07:01.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:07:01.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:07:02.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 66030592 bytes 2026-02-22T20:07:02.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 91932672 bytes 2026-02-22T20:07:04.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 125915136 bytes 2026-02-22T20:07:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 159750144 bytes 2026-02-22T20:07:06.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 221591552 bytes 2026-02-22T20:07:06.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 258892800 bytes 2026-02-22T20:07:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 324221952 bytes 2026-02-22T20:07:08.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 417369088 bytes 2026-02-22T20:07:09.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 480097280 bytes 2026-02-22T20:07:09.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 501015552 bytes 2026-02-22T20:07:10.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 536970240 bytes 2026-02-22T20:07:10.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 577603584 bytes 2026-02-22T20:07:11.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 634474496 bytes 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_2 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 journal 128M 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:12.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_2 128M 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:12.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_2 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_2 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:07:12.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:07:12.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=8 2026-02-22T20:07:12.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:07:12.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:07:13.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 51398656 bytes 2026-02-22T20:07:13.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 110624768 bytes 2026-02-22T20:07:14.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 163513344 bytes 2026-02-22T20:07:14.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 185092096 bytes 2026-02-22T20:07:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 213339136 bytes 2026-02-22T20:07:16.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 227495936 bytes 2026-02-22T20:07:17.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 263275520 bytes 2026-02-22T20:07:19.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 391759872 bytes 2026-02-22T20:07:20.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_3 2026-02-22T20:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 journal 128M 2026-02-22T20:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_3 128M 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:07:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:07:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:07:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:07:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_3 2026-02-22T20:07:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_3 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:07:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:07:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T20:07:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:07:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:07:21.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 81752064 bytes 2026-02-22T20:07:23.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 151170048 bytes 2026-02-22T20:07:24.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 166208512 bytes 2026-02-22T20:07:25.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 207932416 bytes 2026-02-22T20:07:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 269534208 bytes 2026-02-22T20:07:26.220 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5881638116919484) > weight (0.25) 2026-02-22T20:07:26.220 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-22T20:07:26.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-22T20:07:26.220 INFO:tasks.rbd_mirror_thrash:waiting for 2.562686701739122 secs before reviving daemons 2026-02-22T20:07:26.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:07:26.207+0000 7f6b384bf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 80767) UID: 1000 2026-02-22T20:07:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 292422656 bytes 2026-02-22T20:07:27.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 339719168 bytes 2026-02-22T20:07:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 382357504 bytes 2026-02-22T20:07:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 391470080 bytes 2026-02-22T20:07:28.784 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-22T20:07:28.784 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:07:28.784 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-22T20:07:28.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-22T20:07:28.784 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-22T20:07:28.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-22T20:07:28.787 INFO:tasks.rbd_mirror_thrash:waiting for 74.09434181738233 secs before thrashing 2026-02-22T20:07:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 406886400 bytes 2026-02-22T20:07:29.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.257701665140028) > weight (0.25) 2026-02-22T20:07:29.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3180131420168445) > weight (0.25) 2026-02-22T20:07:29.719 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2026-02-22T20:07:29.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2026-02-22T20:07:29.720 INFO:tasks.rbd_mirror_thrash:waiting for 9.666517297959187 secs before reviving daemons 2026-02-22T20:07:29.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:07:29.703+0000 7f4acecb9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 20106) UID: 1000 2026-02-22T20:07:30.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 449838080 bytes 2026-02-22T20:07:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 471308288 bytes 2026-02-22T20:07:31.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 549531648 bytes 2026-02-22T20:07:32.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 560746496 bytes 2026-02-22T20:07:32.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:07:32.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:07:32.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_4 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 journal 128M 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_4 128M 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:32.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_4 2026-02-22T20:07:32.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:32.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:32.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:07:32.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:07:32.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:07:32.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:07:32.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_4 2026-02-22T20:07:32.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:32.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:32.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:07:32.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:07:32.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=8 2026-02-22T20:07:32.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:07:32.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:07:33.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 28251136 bytes 2026-02-22T20:07:34.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 61490176 bytes 2026-02-22T20:07:34.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 85395456 bytes 2026-02-22T20:07:35.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 102220800 bytes 2026-02-22T20:07:36.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 161297408 bytes 2026-02-22T20:07:37.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 234767360 bytes 2026-02-22T20:07:39.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 267798528 bytes 2026-02-22T20:07:39.388 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2026-02-22T20:07:39.388 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:07:39.388 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2026-02-22T20:07:39.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2026-02-22T20:07:39.388 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2026-02-22T20:07:39.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2026-02-22T20:07:39.390 INFO:tasks.rbd_mirror_thrash:waiting for 86.39685250778588 secs before thrashing 2026-02-22T20:07:40.066 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:07:40.067+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_5 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 journal 128M 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_5 128M 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:07:40.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:07:40.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:07:40.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:40.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_5 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_5 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:07:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:07:40.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T20:07:40.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:07:40.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:07:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 42605568 bytes 2026-02-22T20:07:42.126 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:07:42.127+0000 7f65e934d640 -1 reset not still connected to 0x55f6c7647450 2026-02-22T20:07:42.126 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:07:42.127+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:07:42.127 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:07:42.127+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:07:42.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 142573568 bytes 2026-02-22T20:07:43.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 174140416 bytes 2026-02-22T20:07:44.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 243906560 bytes 2026-02-22T20:07:44.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 249310208 bytes 2026-02-22T20:07:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 282359808 bytes 2026-02-22T20:07:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:07:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:07:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:07:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:07:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:07:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_6 2026-02-22T20:07:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 journal 128M 2026-02-22T20:07:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_6 128M 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_6 2026-02-22T20:07:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:07:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:07:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:07:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:07:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_6 2026-02-22T20:07:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:45.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:07:45.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:07:45.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T20:07:45.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:07:45.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:07:46.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 40720384 bytes 2026-02-22T20:07:48.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 134981632 bytes 2026-02-22T20:07:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 156469248 bytes 2026-02-22T20:07:50.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 168464384 bytes 2026-02-22T20:07:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 175432704 bytes 2026-02-22T20:07:51.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 200523776 bytes 2026-02-22T20:07:51.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 219683840 bytes 2026-02-22T20:07:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 264830976 bytes 2026-02-22T20:07:52.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 282551296 bytes 2026-02-22T20:07:53.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 337845248 bytes 2026-02-22T20:07:53.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 368438272 bytes 2026-02-22T20:07:54.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 392877056 bytes 2026-02-22T20:07:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 420653056 bytes 2026-02-22T20:07:55.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 490232832 bytes 2026-02-22T20:07:55.963 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:07:55.963+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0961a0 2026-02-22T20:07:55.964 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:07:55.963+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:07:55.965 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:07:55.963+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:07:55.965 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:07:55.963+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:07:55.965 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:07:55.963+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:07:55.965 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:07:55.963+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:07:55.965 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:07:55.963+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:07:56.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 524590080 bytes 2026-02-22T20:07:57.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_7 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 journal 128M 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:57.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_7 128M 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:07:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:57.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:07:57.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:07:57.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:07:57.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:07:57.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_7 2026-02-22T20:07:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:07:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:07:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:07:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:07:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_7 2026-02-22T20:07:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:07:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:07:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:07:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:07:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=10 2026-02-22T20:07:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:07:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:07:58.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 59036672 bytes 2026-02-22T20:07:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 97731584 bytes 2026-02-22T20:08:00.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 156405760 bytes 2026-02-22T20:08:01.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 201574400 bytes 2026-02-22T20:08:02.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 238522368 bytes 2026-02-22T20:08:03.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 294816768 bytes 2026-02-22T20:08:05.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 385991680 bytes 2026-02-22T20:08:07.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 479808512 bytes 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_8 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 journal 128M 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_8 128M 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:08:07.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:08:07.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:07.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_8 2026-02-22T20:08:07.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:08:07.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:07.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:08:07.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:08:07.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:08:07.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:08:07.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_8 2026-02-22T20:08:07.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:07.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:08:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:08:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=13 2026-02-22T20:08:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:08:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:08:08.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 40080384 bytes 2026-02-22T20:08:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 45096960 bytes 2026-02-22T20:08:09.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 45720576 bytes 2026-02-22T20:08:09.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 103446528 bytes 2026-02-22T20:08:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 150777856 bytes 2026-02-22T20:08:12.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 198354944 bytes 2026-02-22T20:08:12.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 244900864 bytes 2026-02-22T20:08:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 249773056 bytes 2026-02-22T20:08:13.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 305074176 bytes 2026-02-22T20:08:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 327170048 bytes 2026-02-22T20:08:15.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 398279680 bytes 2026-02-22T20:08:15.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 427062272 bytes 2026-02-22T20:08:16.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 496684032 bytes 2026-02-22T20:08:17.408 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:17.406+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0961a0 2026-02-22T20:08:17.408 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:17.406+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:08:17.409 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:17.406+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:08:17.409 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:17.406+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:08:17.409 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:17.406+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:08:17.410 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:17.410+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:08:17.410 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:17.410+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:08:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 513795072 bytes 2026-02-22T20:08:18.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 565915648 bytes 2026-02-22T20:08:18.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 577639424 bytes 2026-02-22T20:08:19.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1700 IOs, 619622400 bytes 2026-02-22T20:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_9 2026-02-22T20:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 journal 128M 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_9 128M 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_9 2026-02-22T20:08:20.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:08:20.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:20.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:08:20.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:08:20.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:08:20.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:08:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_9 2026-02-22T20:08:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:08:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:08:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=10 2026-02-22T20:08:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:08:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:08:22.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 73306112 bytes 2026-02-22T20:08:24.138 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:24.138+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:08:24.138 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:24.138+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:08:24.138 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:24.138+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:08:24.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 138847232 bytes 2026-02-22T20:08:25.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 213199872 bytes 2026-02-22T20:08:26.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 219625472 bytes 2026-02-22T20:08:26.339 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:08:26.338+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:08:26.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 228502528 bytes 2026-02-22T20:08:26.807 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:26.806+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:08:26.808 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:26.806+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:08:26.808 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:26.806+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:08:27.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 279339008 bytes 2026-02-22T20:08:27.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 293683200 bytes 2026-02-22T20:08:28.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 347597824 bytes 2026-02-22T20:08:29.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 395976704 bytes 2026-02-22T20:08:29.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 412479488 bytes 2026-02-22T20:08:29.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 454187008 bytes 2026-02-22T20:08:30.029 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:30.026+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:08:30.029 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:30.026+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:08:30.029 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:30.026+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:08:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:08:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:08:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:08:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:08:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:08:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_10 2026-02-22T20:08:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 journal 128M 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_10 128M 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:08:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:08:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:08:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:08:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_10 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_10 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:08:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:08:30.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=8 2026-02-22T20:08:30.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:08:30.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:08:31.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 40343552 bytes 2026-02-22T20:08:33.658 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:08:33.658+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:08:33.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 173998080 bytes 2026-02-22T20:08:34.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 237206528 bytes 2026-02-22T20:08:35.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 281330688 bytes 2026-02-22T20:08:36.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 373961728 bytes 2026-02-22T20:08:37.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 415800320 bytes 2026-02-22T20:08:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:08:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:08:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_11 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 journal 128M 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_11 128M 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:08:38.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:38.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_11 2026-02-22T20:08:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:08:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:08:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:08:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:08:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:08:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_11 2026-02-22T20:08:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:08:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:08:38.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=6 2026-02-22T20:08:38.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:08:38.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:08:39.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 21102592 bytes 2026-02-22T20:08:40.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 43155456 bytes 2026-02-22T20:08:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 94408704 bytes 2026-02-22T20:08:41.919 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:08:41.917+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:08:42.882 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7166972157903934) > weight (0.25) 2026-02-22T20:08:42.883 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8864167289756095) > weight (0.25) 2026-02-22T20:08:42.883 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8746026747858868) > weight (0.25) 2026-02-22T20:08:42.883 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8424782108739768) > weight (0.25) 2026-02-22T20:08:42.883 INFO:tasks.rbd_mirror_thrash:waiting for 109.70079099068482 secs before thrashing 2026-02-22T20:08:42.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 159559680 bytes 2026-02-22T20:08:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 180829184 bytes 2026-02-22T20:08:43.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 184260608 bytes 2026-02-22T20:08:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 227291136 bytes 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_12 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 journal 128M 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_12 128M 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_12 2026-02-22T20:08:45.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:08:45.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:45.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:08:45.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:08:45.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:08:45.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:08:45.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_12 2026-02-22T20:08:45.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:45.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:45.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:08:45.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:08:45.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=14 2026-02-22T20:08:45.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:08:45.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:08:45.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 42374144 bytes 2026-02-22T20:08:46.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 81774592 bytes 2026-02-22T20:08:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 82415616 bytes 2026-02-22T20:08:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 83042304 bytes 2026-02-22T20:08:48.885 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:48.881+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:08:48.885 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:48.881+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:08:48.885 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:48.881+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:08:49.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 184645632 bytes 2026-02-22T20:08:49.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 204084224 bytes 2026-02-22T20:08:49.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 224595968 bytes 2026-02-22T20:08:49.820 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:08:49.802+0000 7f557ffc8640 -1 reset not still connected to 0x55923575f2b0 2026-02-22T20:08:50.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 261607424 bytes 2026-02-22T20:08:50.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 291291136 bytes 2026-02-22T20:08:51.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 314370048 bytes 2026-02-22T20:08:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 365821952 bytes 2026-02-22T20:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 431042560 bytes 2026-02-22T20:08:54.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 498382848 bytes 2026-02-22T20:08:54.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 523780096 bytes 2026-02-22T20:08:55.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 568299520 bytes 2026-02-22T20:08:56.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 608230400 bytes 2026-02-22T20:08:58.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1700 IOs, 667948032 bytes 2026-02-22T20:08:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1800 IOs, 688192512 bytes 2026-02-22T20:08:59.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:08:59.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:08:59.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_13 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 journal 128M 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_13 128M 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:08:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:08:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_13 2026-02-22T20:08:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:08:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:08:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:08:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:08:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:08:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_13 2026-02-22T20:08:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:08:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:08:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:08:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:08:59.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=10 2026-02-22T20:08:59.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:08:59.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:08:59.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 605184 bytes 2026-02-22T20:08:59.497 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:59.493+0000 7f8da6485640 -1 reset not still connected to 0x55abf95c4270 2026-02-22T20:08:59.518 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:59.517+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:08:59.519 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:59.517+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:08:59.519 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:59.517+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:08:59.650 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:08:59.649+0000 7f8da6485640 -1 reset not still connected to 0x55abf95c4270 2026-02-22T20:09:00.353 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:00.349+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:09:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 47776768 bytes 2026-02-22T20:09:01.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 73517056 bytes 2026-02-22T20:09:03.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 151656448 bytes 2026-02-22T20:09:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 178588672 bytes 2026-02-22T20:09:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 202975232 bytes 2026-02-22T20:09:04.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 224432128 bytes 2026-02-22T20:09:04.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 225045504 bytes 2026-02-22T20:09:05.565 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:05.561+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:09:05.787 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-22T20:09:05.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-22T20:09:05.788 INFO:tasks.rbd_mirror_thrash:waiting for 3.7891918401021716 secs before reviving daemons 2026-02-22T20:09:05.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:09:05.770+0000 7f2316035640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 83029) UID: 1000 2026-02-22T20:09:05.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 285903872 bytes 2026-02-22T20:09:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 340614144 bytes 2026-02-22T20:09:07.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 364978176 bytes 2026-02-22T20:09:07.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 379811840 bytes 2026-02-22T20:09:08.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 400421888 bytes 2026-02-22T20:09:08.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 426699776 bytes 2026-02-22T20:09:08.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 433650688 bytes 2026-02-22T20:09:09.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:09:09.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:09:09.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:09:09.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:09:09.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:09:09.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_14 2026-02-22T20:09:09.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 journal 128M 2026-02-22T20:09:09.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:09.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:09.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:09.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:09.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:09:09.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:09.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:09:09.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:09:09.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_14 128M 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_14 2026-02-22T20:09:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:09:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:09:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:09:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:09:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:09:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_14 2026-02-22T20:09:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:09:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:09:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T20:09:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:09:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:09:09.578 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-22T20:09:09.579 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:09:09.579 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-22T20:09:09.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-22T20:09:09.579 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-22T20:09:09.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-22T20:09:09.582 INFO:tasks.rbd_mirror_thrash:waiting for 106.49517121474014 secs before thrashing 2026-02-22T20:09:10.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 58063872 bytes 2026-02-22T20:09:10.191 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:10.189+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3332c0d0 2026-02-22T20:09:10.192 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:10.189+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3ea69ba0 2026-02-22T20:09:10.192 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:10.189+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc1110 2026-02-22T20:09:10.192 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:10.189+0000 7f5e092a6640 -1 reset not still connected to 0x55ac515aba00 2026-02-22T20:09:10.192 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:10.189+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:09:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 117718016 bytes 2026-02-22T20:09:12.509 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:12.505+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:09:12.509 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:12.505+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:09:12.510 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:12.509+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:09:12.510 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:12.509+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:09:12.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 160350208 bytes 2026-02-22T20:09:12.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 179335168 bytes 2026-02-22T20:09:13.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 221368320 bytes 2026-02-22T20:09:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_15 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 journal 128M 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_15 128M 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_15 2026-02-22T20:09:14.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:09:14.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:14.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:09:14.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:09:14.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:09:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:09:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_15 2026-02-22T20:09:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:09:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:09:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=9 2026-02-22T20:09:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:09:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:09:14.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 17867776 bytes 2026-02-22T20:09:16.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 130296832 bytes 2026-02-22T20:09:17.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 150587392 bytes 2026-02-22T20:09:18.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 222526464 bytes 2026-02-22T20:09:19.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 272252928 bytes 2026-02-22T20:09:20.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 320136192 bytes 2026-02-22T20:09:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 355746816 bytes 2026-02-22T20:09:22.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 424650752 bytes 2026-02-22T20:09:23.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 467573760 bytes 2026-02-22T20:09:23.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:09:23.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:09:23.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:09:23.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_16 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 journal 128M 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_16 128M 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_16 2026-02-22T20:09:23.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_16 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:09:23.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:09:23.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=11 2026-02-22T20:09:23.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:09:23.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:09:24.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 33114112 bytes 2026-02-22T20:09:24.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 75265024 bytes 2026-02-22T20:09:25.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 125785088 bytes 2026-02-22T20:09:27.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 168614912 bytes 2026-02-22T20:09:28.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 215227392 bytes 2026-02-22T20:09:28.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 256939008 bytes 2026-02-22T20:09:30.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 320524288 bytes 2026-02-22T20:09:30.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 371554304 bytes 2026-02-22T20:09:31.838 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:31.836+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:09:31.840 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:31.836+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:09:31.840 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:31.836+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:09:31.840 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:31.836+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:09:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 468536320 bytes 2026-02-22T20:09:33.043 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:33.040+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:09:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 579099648 bytes 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_17 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 journal 128M 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:34.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_17 128M 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_17 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_17 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:09:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:09:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=8 2026-02-22T20:09:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:09:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:09:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 69015552 bytes 2026-02-22T20:09:36.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 85056512 bytes 2026-02-22T20:09:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 139028480 bytes 2026-02-22T20:09:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 199891968 bytes 2026-02-22T20:09:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 267067392 bytes 2026-02-22T20:09:40.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 286824448 bytes 2026-02-22T20:09:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 340093952 bytes 2026-02-22T20:09:42.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 371956736 bytes 2026-02-22T20:09:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:09:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:09:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:09:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:09:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:09:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_18 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 journal 128M 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_18 128M 2026-02-22T20:09:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_18 2026-02-22T20:09:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:09:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:09:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:09:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:09:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:09:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_18 2026-02-22T20:09:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:42.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:09:42.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:09:42.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T20:09:42.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:09:42.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:09:42.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 4158464 bytes 2026-02-22T20:09:44.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 145886208 bytes 2026-02-22T20:09:46.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 168027136 bytes 2026-02-22T20:09:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 184421376 bytes 2026-02-22T20:09:47.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 238926848 bytes 2026-02-22T20:09:47.897 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:47.892+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:09:47.897 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:47.892+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:09:47.897 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:47.892+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:09:48.022 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:48.020+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:09:48.023 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:48.020+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:09:48.024 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:48.020+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:09:48.024 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:48.020+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:09:48.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 256319488 bytes 2026-02-22T20:09:48.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 284897280 bytes 2026-02-22T20:09:49.528 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:49.524+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:09:49.529 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:49.524+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:09:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 372558848 bytes 2026-02-22T20:09:50.308 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:50.304+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:09:50.308 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:50.304+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:09:50.309 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:50.304+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:09:50.309 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:50.304+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:09:50.309 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:50.304+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:09:50.309 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:09:50.304+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:09:50.783 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:50.780+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:09:50.783 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:50.780+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:09:51.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 429281280 bytes 2026-02-22T20:09:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 443579392 bytes 2026-02-22T20:09:52.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 481412096 bytes 2026-02-22T20:09:52.817 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:52.812+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3332c0d0 2026-02-22T20:09:52.817 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:52.812+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3ea69ba0 2026-02-22T20:09:52.817 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:52.812+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc1110 2026-02-22T20:09:52.818 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:52.816+0000 7f5e092a6640 -1 reset not still connected to 0x55ac515aba00 2026-02-22T20:09:52.818 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:52.816+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:09:52.818 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:52.816+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:09:53.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 553902080 bytes 2026-02-22T20:09:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:09:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:09:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:09:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:09:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:09:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_19 2026-02-22T20:09:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 journal 128M 2026-02-22T20:09:54.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_19 128M 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:09:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:09:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:09:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:09:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:09:54.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_19 2026-02-22T20:09:54.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_19 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:09:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:09:54.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=11 2026-02-22T20:09:54.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:09:54.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:09:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 12512256 bytes 2026-02-22T20:09:55.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 21456896 bytes 2026-02-22T20:09:55.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 44164096 bytes 2026-02-22T20:09:58.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 202758144 bytes 2026-02-22T20:09:59.195 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:59.191+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0961a0 2026-02-22T20:09:59.197 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:59.191+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:09:59.197 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:59.191+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:09:59.197 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:59.191+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:09:59.197 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:59.191+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:09:59.197 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:59.191+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:09:59.198 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:59.195+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:09:59.198 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:59.195+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:09:59.198 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:09:59.195+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:09:59.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 248240128 bytes 2026-02-22T20:10:00.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 263140352 bytes 2026-02-22T20:10:00.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 288147456 bytes 2026-02-22T20:10:01.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 335981568 bytes 2026-02-22T20:10:01.835 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:01.831+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:10:01.836 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:01.831+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:10:01.836 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:01.831+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:10:02.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 375227392 bytes 2026-02-22T20:10:02.488 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:10:02.483+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:10:02.489 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:10:02.483+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:10:02.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 402209792 bytes 2026-02-22T20:10:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 451998720 bytes 2026-02-22T20:10:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 460907520 bytes 2026-02-22T20:10:05.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 506134528 bytes 2026-02-22T20:10:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:10:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:10:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:10:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:10:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:10:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_20 2026-02-22T20:10:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 journal 128M 2026-02-22T20:10:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:05.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_20 128M 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_20 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_20 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:10:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:10:05.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=6 2026-02-22T20:10:05.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:10:05.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:10:06.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 70047744 bytes 2026-02-22T20:10:07.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 101643264 bytes 2026-02-22T20:10:07.758 INFO:tasks.ceph.cluster2.osd.0.trial066.stderr:2026-02-22T20:10:07.741+0000 7f21cdd98640 -1 reset not still connected to 0x5578dae811e0 2026-02-22T20:10:09.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 154103808 bytes 2026-02-22T20:10:09.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 216679424 bytes 2026-02-22T20:10:10.569 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:10.567+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:10:10.569 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:10.567+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2f311450 2026-02-22T20:10:10.570 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:10.567+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3332c0d0 2026-02-22T20:10:10.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 250059776 bytes 2026-02-22T20:10:11.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 288258048 bytes 2026-02-22T20:10:11.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 304412672 bytes 2026-02-22T20:10:11.631 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:11.631+0000 7f5e092a6640 -1 reset not still connected to 0x55ac565e0ea0 2026-02-22T20:10:11.632 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:11.631+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:10:11.632 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:11.631+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:10:11.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 307444736 bytes 2026-02-22T20:10:11.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:10:11.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_21 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 journal 128M 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:10:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_21 128M 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:10:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:10:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:11.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_21 2026-02-22T20:10:11.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:10:11.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:11.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:10:11.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:10:11.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:10:11.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:10:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_21 2026-02-22T20:10:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:10:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:10:11.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T20:10:11.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:10:11.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:10:13.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 65882112 bytes 2026-02-22T20:10:13.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 68315136 bytes 2026-02-22T20:10:13.427 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:10:13.427+0000 7f8da6485640 -1 reset not still connected to 0x55abf96e7040 2026-02-22T20:10:13.427 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:10:13.427+0000 7f8da6485640 -1 reset not still connected to 0x55abf9739860 2026-02-22T20:10:13.608 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:10:13.607+0000 7f8da6485640 -1 reset not still connected to 0x55abf95c4270 2026-02-22T20:10:14.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 136521728 bytes 2026-02-22T20:10:15.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 162677760 bytes 2026-02-22T20:10:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 206944256 bytes 2026-02-22T20:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 254683136 bytes 2026-02-22T20:10:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 268476416 bytes 2026-02-22T20:10:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 292348928 bytes 2026-02-22T20:10:18.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 316176384 bytes 2026-02-22T20:10:18.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 316797952 bytes 2026-02-22T20:10:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 320960512 bytes 2026-02-22T20:10:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 344675328 bytes 2026-02-22T20:10:20.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 374661120 bytes 2026-02-22T20:10:20.490 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:20.491+0000 7f5e092a6640 -1 reset not still connected to 0x55ac56c26680 2026-02-22T20:10:20.492 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:20.491+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5c32f6c0 2026-02-22T20:10:20.492 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:20.491+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cea9040 2026-02-22T20:10:20.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 390985728 bytes 2026-02-22T20:10:21.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 449743872 bytes 2026-02-22T20:10:22.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 467152896 bytes 2026-02-22T20:10:23.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1700 IOs, 552509440 bytes 2026-02-22T20:10:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:10:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:10:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:10:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:10:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:10:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_22 2026-02-22T20:10:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 journal 128M 2026-02-22T20:10:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_22 128M 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:10:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:10:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:10:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:10:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_22 2026-02-22T20:10:23.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:10:23.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:10:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:10:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:10:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:10:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_22 2026-02-22T20:10:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:10:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:10:23.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T20:10:23.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:10:23.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:10:23.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 4275200 bytes 2026-02-22T20:10:24.302 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:24.303+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:10:25.314 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.315+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:10:25.314 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.315+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:10:25.315 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.315+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:10:25.315 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.315+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:10:25.423 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.423+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:10:25.424 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.423+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:10:25.424 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.423+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:10:25.424 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.423+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:10:25.449 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.447+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:10:25.450 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.447+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:10:25.450 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.447+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:10:25.450 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.447+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:10:25.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 103986176 bytes 2026-02-22T20:10:25.866 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:25.867+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2eed0d00 2026-02-22T20:10:25.969 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.967+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:10:25.970 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.967+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:10:25.970 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.967+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:10:25.970 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:25.967+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:10:28.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 183836672 bytes 2026-02-22T20:10:28.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:10:28.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:10:28.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:10:28.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:10:28.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_23 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 journal 128M 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:10:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_23 128M 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_23 2026-02-22T20:10:28.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:10:28.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_23 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=7 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:10:29.208 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:29.207+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:10:29.209 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:29.207+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:10:29.209 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:29.207+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:10:29.209 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:29.207+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:10:29.210 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:29.207+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:10:29.210 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:29.207+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:10:29.210 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:29.207+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:10:29.210 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:29.211+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:10:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 26759168 bytes 2026-02-22T20:10:29.612 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:29.611+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:10:29.613 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:29.611+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:10:29.613 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:29.611+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:10:29.614 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:29.611+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:10:29.614 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:29.615+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:10:29.614 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:29.615+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:10:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 94979072 bytes 2026-02-22T20:10:32.584 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2026-02-22T20:10:32.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2026-02-22T20:10:32.585 INFO:tasks.rbd_mirror_thrash:waiting for 4.366948219885217 secs before reviving daemons 2026-02-22T20:10:32.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:10:32.566+0000 7f1953ad3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 75626) UID: 1000 2026-02-22T20:10:32.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:10:32.594+0000 7f194cac5640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x56107d849880 handle_process_entry_safe: failed to commit journal event: (85) Interrupted system call should be restarted 2026-02-22T20:10:32.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:10:32.594+0000 7f194cac5640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x56107d849880 handle_replay_complete: replay encountered an error: (85) Interrupted system call should be restarted 2026-02-22T20:10:32.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 137229312 bytes 2026-02-22T20:10:33.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 158401536 bytes 2026-02-22T20:10:33.279 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:33.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:10:33.279 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:33.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:10:33.280 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:33.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:10:33.280 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:33.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:10:33.280 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:33.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:10:33.280 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:33.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:10:33.282 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:33.283+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:10:34.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 249108480 bytes 2026-02-22T20:10:35.277 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.275+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2d126340 2026-02-22T20:10:35.278 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2d126820 2026-02-22T20:10:35.279 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0961a0 2026-02-22T20:10:35.279 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:10:35.280 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:10:35.280 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:10:35.280 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:10:35.281 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:10:35.281 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:10:35.281 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.279+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:10:35.281 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:35.283+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:10:35.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:10:35.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_24 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 journal 128M 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_24 128M 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:35.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:10:35.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:35.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:10:35.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:10:35.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:10:35.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:35.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_24 2026-02-22T20:10:35.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:10:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:10:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:10:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:10:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:10:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_24 2026-02-22T20:10:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:10:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:10:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=10 2026-02-22T20:10:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:10:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:10:36.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 19633152 bytes 2026-02-22T20:10:36.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 49092608 bytes 2026-02-22T20:10:36.953 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2026-02-22T20:10:36.953 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:10:36.954 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2026-02-22T20:10:36.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-22T20:10:36.954 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-22T20:10:36.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-22T20:10:36.956 INFO:tasks.rbd_mirror_thrash:waiting for 109.7786706792142 secs before thrashing 2026-02-22T20:10:37.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 95141888 bytes 2026-02-22T20:10:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 118153216 bytes 2026-02-22T20:10:39.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 212835328 bytes 2026-02-22T20:10:40.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 238021632 bytes 2026-02-22T20:10:40.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 238645248 bytes 2026-02-22T20:10:40.966 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:40.966+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:10:40.966 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:40.966+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:10:40.967 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:40.966+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:10:40.967 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:40.966+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:10:40.967 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:40.966+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:10:40.968 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:40.966+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:10:40.969 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:40.966+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:10:40.969 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:40.966+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:10:40.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 265999360 bytes 2026-02-22T20:10:41.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 273607680 bytes 2026-02-22T20:10:42.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 348302336 bytes 2026-02-22T20:10:43.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 373822464 bytes 2026-02-22T20:10:43.926 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:10:43.906+0000 7f557ffc8640 -1 reset not still connected to 0x5592036511e0 2026-02-22T20:10:43.926 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:10:43.906+0000 7f557ffc8640 -1 reset not still connected to 0x559203f9add0 2026-02-22T20:10:43.972 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:43.970+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:10:43.974 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:43.974+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:10:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 521546752 bytes 2026-02-22T20:10:45.107 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:45.106+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:10:45.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 552403968 bytes 2026-02-22T20:10:45.533 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:45.518+0000 7f65e934d640 -1 reset not still connected to 0x55f6de693860 2026-02-22T20:10:45.533 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:45.522+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0391ee0 2026-02-22T20:10:45.533 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:45.522+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:10:45.533 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:10:45.522+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:10:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:10:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:10:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:10:45.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_25 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 journal 128M 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_25 128M 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_25 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_25 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:10:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:10:45.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T20:10:45.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:10:45.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:10:46.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 4879360 bytes 2026-02-22T20:10:47.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 79077376 bytes 2026-02-22T20:10:47.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:10:47.074+0000 7f98c9b77640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x559db4548540 finish: resending after timeout 2026-02-22T20:10:48.718 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:48.718+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:10:48.719 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:10:48.718+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:10:48.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 144275456 bytes 2026-02-22T20:10:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 151794688 bytes 2026-02-22T20:10:49.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 164706304 bytes 2026-02-22T20:10:49.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 165325824 bytes 2026-02-22T20:10:50.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 207657984 bytes 2026-02-22T20:10:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:10:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:10:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:10:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:10:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:10:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_26 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 journal 128M 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_26 128M 2026-02-22T20:10:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_26 2026-02-22T20:10:51.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:10:51.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:51.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:10:51.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:10:51.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:10:51.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:10:51.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_26 2026-02-22T20:10:51.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:51.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:51.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:10:51.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:10:51.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T20:10:51.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:10:51.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:10:51.266 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:10:51.247+0000 7f557ffc8640 -1 reset not still connected to 0x559220202750 2026-02-22T20:10:52.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 57737216 bytes 2026-02-22T20:10:52.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:10:52.819+0000 7f98c9b77640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x559db4548540 finish: resending after timeout 2026-02-22T20:10:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 79590400 bytes 2026-02-22T20:10:54.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 122864640 bytes 2026-02-22T20:10:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 148391936 bytes 2026-02-22T20:10:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_27 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 journal 128M 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_27 128M 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:10:56.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:10:56.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:10:56.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:10:56.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_27 2026-02-22T20:10:56.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:10:56.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:56.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:10:56.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:10:56.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:10:56.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:10:56.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_27 2026-02-22T20:10:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:10:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:10:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:10:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:10:56.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=9 2026-02-22T20:10:56.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:10:56.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:10:56.078 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-22T20:10:56.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-22T20:10:56.079 INFO:tasks.rbd_mirror_thrash:waiting for 1.0571716796124042 secs before reviving daemons 2026-02-22T20:10:56.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:10:56.063+0000 7f478ee87640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 85125) UID: 1000 2026-02-22T20:10:57.137 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-22T20:10:57.137 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:10:57.138 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-22T20:10:57.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-22T20:10:57.138 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-22T20:10:57.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-22T20:10:57.140 INFO:tasks.rbd_mirror_thrash:waiting for 117.21449110496505 secs before thrashing 2026-02-22T20:10:57.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 58676224 bytes 2026-02-22T20:10:58.398 INFO:tasks.ceph.cluster2.osd.0.trial066.stderr:2026-02-22T20:10:58.379+0000 7f21cdd98640 -1 reset not still connected to 0x5578d9000dd0 2026-02-22T20:10:58.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 140414976 bytes 2026-02-22T20:10:59.400 INFO:tasks.ceph.cluster2.osd.0.trial066.stderr:2026-02-22T20:10:59.383+0000 7f21cdd98640 -1 reset not still connected to 0x5578d005a270 2026-02-22T20:10:59.400 INFO:tasks.ceph.cluster2.osd.0.trial066.stderr:2026-02-22T20:10:59.383+0000 7f21cdd98640 -1 reset not still connected to 0x5578da34b6c0 2026-02-22T20:11:00.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 187842560 bytes 2026-02-22T20:11:00.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 224437248 bytes 2026-02-22T20:11:01.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 256104448 bytes 2026-02-22T20:11:01.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 323423232 bytes 2026-02-22T20:11:03.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 354971648 bytes 2026-02-22T20:11:03.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 364121088 bytes 2026-02-22T20:11:03.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 383181824 bytes 2026-02-22T20:11:04.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 404325376 bytes 2026-02-22T20:11:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 416156672 bytes 2026-02-22T20:11:05.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:11:05.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:11:05.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:11:05.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:11:05.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_28 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 journal 128M 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_28 128M 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_28 2026-02-22T20:11:05.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:11:05.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:05.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:11:05.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:11:05.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:11:05.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:11:05.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_28 2026-02-22T20:11:05.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:05.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:05.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:11:05.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:11:05.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=7 2026-02-22T20:11:05.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:11:05.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:11:05.172 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:05.170+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:05.172 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:05.170+0000 7f8da6485640 -1 reset not still connected to 0x55abeafd8820 2026-02-22T20:11:05.173 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:05.170+0000 7f8da6485640 -1 reset not still connected to 0x55abeb57d1e0 2026-02-22T20:11:05.173 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:05.170+0000 7f8da6485640 -1 reset not still connected to 0x55abf35e32b0 2026-02-22T20:11:05.173 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:05.170+0000 7f8da6485640 -1 reset not still connected to 0x55abf4df24e0 2026-02-22T20:11:05.173 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:05.170+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 37367808 bytes 2026-02-22T20:11:06.877 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:06.874+0000 7f8da6485640 -1 reset not still connected to 0x55abd9d22000 2026-02-22T20:11:06.877 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:06.874+0000 7f8da6485640 -1 reset not still connected to 0x55abf96e7040 2026-02-22T20:11:06.877 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:06.874+0000 7f8da6485640 -1 reset not still connected to 0x55abf9739860 2026-02-22T20:11:07.271 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:11:07.271 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:11:07.272 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:11:07.272 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:11:07.272 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:11:07.273 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:11:07.273 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:11:07.273 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:11:07.274 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3334e8f0 2026-02-22T20:11:07.274 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5147b380 2026-02-22T20:11:07.274 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.270+0000 7f5e092a6640 -1 reset not still connected to 0x55ac565e0ea0 2026-02-22T20:11:07.274 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.274+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:11:07.275 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.274+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5c18c4e0 2026-02-22T20:11:07.275 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.274+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:11:07.504 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.502+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:11:07.505 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.502+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:11:07.506 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.506+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:11:07.506 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.506+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:11:07.527 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:07.526+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:07.778 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:07.778+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:11:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 98361344 bytes 2026-02-22T20:11:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 153268224 bytes 2026-02-22T20:11:09.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 170957824 bytes 2026-02-22T20:11:11.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 247247872 bytes 2026-02-22T20:11:11.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 282134528 bytes 2026-02-22T20:11:12.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:11:12.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:11:12.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:11:12.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:11:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:11:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_29 2026-02-22T20:11:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 journal 128M 2026-02-22T20:11:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:11:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:11:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:11:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_29 128M 2026-02-22T20:11:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:11:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:11:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:11:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:11:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_29 2026-02-22T20:11:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:11:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:11:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:11:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:11:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:11:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_29 2026-02-22T20:11:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:11:12.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:11:12.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T20:11:12.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:11:12.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:11:12.244 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T20:11:12.227+0000 7fdfce879640 -1 reset not still connected to 0x555a8c053e10 2026-02-22T20:11:12.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 17310720 bytes 2026-02-22T20:11:12.505 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:12.502+0000 7f8da6485640 -1 reset not still connected to 0x55abeb57d1e0 2026-02-22T20:11:12.505 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:12.502+0000 7f8da6485640 -1 reset not still connected to 0x55abf35e32b0 2026-02-22T20:11:12.506 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:12.506+0000 7f8da6485640 -1 reset not still connected to 0x55abf4df24e0 2026-02-22T20:11:12.506 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:12.506+0000 7f8da6485640 -1 reset not still connected to 0x55abf4e1f450 2026-02-22T20:11:12.506 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:12.506+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:12.603 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:12.602+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0961a0 2026-02-22T20:11:12.604 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:12.602+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:11:12.604 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:12.602+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:11:12.604 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:12.602+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:11:12.604 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:12.602+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:11:12.605 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:12.602+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:11:12.605 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:12.602+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:11:12.605 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:12.606+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:11:12.606 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:12.606+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:11:12.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 48745472 bytes 2026-02-22T20:11:13.228 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:11:13.211+0000 7f557ffc8640 -1 reset not still connected to 0x559205fd5e10 2026-02-22T20:11:13.228 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:11:13.211+0000 7f557ffc8640 -1 reset not still connected to 0x55921eecf790 2026-02-22T20:11:13.228 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:11:13.211+0000 7f557ffc8640 -1 reset not still connected to 0x55923a2e7790 2026-02-22T20:11:13.228 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:11:13.211+0000 7f557ffc8640 -1 reset not still connected to 0x559205fd5e10 2026-02-22T20:11:13.228 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:11:13.211+0000 7f557ffc8640 -1 reset not still connected to 0x55921eecf790 2026-02-22T20:11:13.228 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:11:13.211+0000 7f557ffc8640 -1 reset not still connected to 0x55923a2e7790 2026-02-22T20:11:13.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:11:13.779+0000 7f98c9376640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x559d9e8d03c0 finish: resending after timeout 2026-02-22T20:11:14.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 85723136 bytes 2026-02-22T20:11:15.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 128802816 bytes 2026-02-22T20:11:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 187537408 bytes 2026-02-22T20:11:16.481 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:16.478+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:11:16.481 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:16.478+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:11:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 212756480 bytes 2026-02-22T20:11:16.947 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:16.946+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:11:16.948 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:16.946+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:11:16.948 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:16.946+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:11:16.948 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:16.946+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:11:16.948 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:16.946+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:11:16.949 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:16.946+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:11:17.075 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.074+0000 7f8da6485640 -1 reset not still connected to 0x55abf4df24e0 2026-02-22T20:11:17.076 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.074+0000 7f8da6485640 -1 reset not still connected to 0x55abf4e1f450 2026-02-22T20:11:17.076 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.074+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:17.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 272295936 bytes 2026-02-22T20:11:17.921 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.918+0000 7f8da6485640 -1 reset not still connected to 0x55abc5af1110 2026-02-22T20:11:17.922 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.922+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:11:17.923 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.922+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:11:17.923 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.922+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:11:17.982 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.982+0000 7f8da6485640 -1 reset not still connected to 0x55abeafd8820 2026-02-22T20:11:17.983 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.982+0000 7f8da6485640 -1 reset not still connected to 0x55abeb48b520 2026-02-22T20:11:17.983 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.982+0000 7f8da6485640 -1 reset not still connected to 0x55abeb57d1e0 2026-02-22T20:11:17.984 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.982+0000 7f8da6485640 -1 reset not still connected to 0x55abf35e32b0 2026-02-22T20:11:17.984 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.982+0000 7f8da6485640 -1 reset not still connected to 0x55abf4df24e0 2026-02-22T20:11:17.984 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.982+0000 7f8da6485640 -1 reset not still connected to 0x55abf4e1f450 2026-02-22T20:11:17.984 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:17.982+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:18.232 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:18.230+0000 7f8da6485640 -1 reset not still connected to 0x55abf35e32b0 2026-02-22T20:11:18.233 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:18.230+0000 7f8da6485640 -1 reset not still connected to 0x55abf4df24e0 2026-02-22T20:11:18.233 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:18.230+0000 7f8da6485640 -1 reset not still connected to 0x55abf4e1f450 2026-02-22T20:11:18.235 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:18.234+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 359685120 bytes 2026-02-22T20:11:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 374096896 bytes 2026-02-22T20:11:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 402961408 bytes 2026-02-22T20:11:20.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 454769664 bytes 2026-02-22T20:11:21.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 486276096 bytes 2026-02-22T20:11:23.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 587313152 bytes 2026-02-22T20:11:23.572 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:23.569+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:11:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:11:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:11:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:11:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:11:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:11:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_30 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 journal 128M 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_30 128M 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_30 2026-02-22T20:11:24.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:11:24.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:24.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:11:24.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:11:24.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:11:24.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:11:24.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_30 2026-02-22T20:11:24.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:24.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:24.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:11:24.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:11:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=13 2026-02-22T20:11:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:11:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:11:24.396 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:24.393+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0961a0 2026-02-22T20:11:24.397 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:24.393+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:11:24.397 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:24.393+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:11:24.397 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:24.393+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:11:24.397 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:24.393+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:11:24.398 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:24.397+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:11:24.398 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:24.397+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:11:24.399 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:24.397+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:11:24.399 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:24.397+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:11:25.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 69193728 bytes 2026-02-22T20:11:25.832 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:25.829+0000 7f65e934d640 -1 reset not still connected to 0x55f6de693860 2026-02-22T20:11:25.832 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:25.829+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0391ee0 2026-02-22T20:11:25.832 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:25.829+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:11:25.832 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:25.829+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:11:25.906 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:25.905+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:11:26.089 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:26.085+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:11:26.089 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:26.085+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:11:26.089 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:26.085+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:11:26.089 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:26.089+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:11:26.090 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:26.089+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:11:26.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 143822848 bytes 2026-02-22T20:11:26.757 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:26.753+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:11:27.040 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:27.037+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4b9c65b0 2026-02-22T20:11:27.040 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:27.037+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4ba59c70 2026-02-22T20:11:27.040 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:27.037+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4c358000 2026-02-22T20:11:27.040 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:27.037+0000 7f5e092a6640 -1 reset not still connected to 0x55ac519be1a0 2026-02-22T20:11:27.040 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:27.037+0000 7f5e092a6640 -1 reset not still connected to 0x55ac51ffdd40 2026-02-22T20:11:27.279 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:27.277+0000 7f8da6485640 -1 reset not still connected to 0x55abf4e1f450 2026-02-22T20:11:27.279 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:27.277+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:27.600 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:27.597+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:11:27.600 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:27.597+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:11:27.601 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:27.597+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:11:28.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 166238208 bytes 2026-02-22T20:11:29.118 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:29.117+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:11:29.120 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:29.117+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5c18c4e0 2026-02-22T20:11:29.120 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:29.117+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:11:29.235 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:29.233+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:11:29.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 207677440 bytes 2026-02-22T20:11:30.082 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:30.081+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:11:30.082 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:30.081+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:11:30.082 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:30.081+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:11:30.390 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:30.385+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:11:30.391 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:30.389+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:11:30.391 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:30.389+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:11:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 275659776 bytes 2026-02-22T20:11:30.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 289577984 bytes 2026-02-22T20:11:31.631 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:31.629+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:31.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 379379712 bytes 2026-02-22T20:11:32.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 411843584 bytes 2026-02-22T20:11:32.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 436865024 bytes 2026-02-22T20:11:33.277 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:33.273+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:33.394 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:33.389+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:33.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 493378560 bytes 2026-02-22T20:11:33.835 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:33.833+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0961a0 2026-02-22T20:11:33.836 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:33.833+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:11:33.836 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:33.833+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:11:33.836 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:33.833+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:11:33.836 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:33.833+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:11:33.837 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:33.833+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:11:33.837 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:33.833+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:11:33.838 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:33.837+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:11:33.838 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:33.837+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:11:34.350 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:34.349+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:11:34.352 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:34.349+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:11:34.352 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:34.349+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:11:34.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 524609536 bytes 2026-02-22T20:11:34.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 525239296 bytes 2026-02-22T20:11:35.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 543887360 bytes 2026-02-22T20:11:36.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 602126336 bytes 2026-02-22T20:11:36.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 652155904 bytes 2026-02-22T20:11:37.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_31 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 journal 128M 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:37.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_31 128M 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:37.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:11:37.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:11:37.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:11:37.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:37.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_31 2026-02-22T20:11:37.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:11:37.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:37.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:11:37.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:11:37.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:11:37.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:11:37.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_31 2026-02-22T20:11:37.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:37.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:37.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:11:37.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:11:37.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=10 2026-02-22T20:11:37.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:11:37.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:11:37.348 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:37.345+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:11:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 2289664 bytes 2026-02-22T20:11:38.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 46273536 bytes 2026-02-22T20:11:38.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 53216256 bytes 2026-02-22T20:11:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 80020480 bytes 2026-02-22T20:11:41.194 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:41.193+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:11:41.194 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:41.193+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:11:41.458 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:41.457+0000 7f8da6485640 -1 reset not still connected to 0x55abf9739860 2026-02-22T20:11:41.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 168035328 bytes 2026-02-22T20:11:41.751 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:41.749+0000 7f8da6485640 -1 reset not still connected to 0x55abe23bd6c0 2026-02-22T20:11:41.751 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:41.749+0000 7f8da6485640 -1 reset not still connected to 0x55abe298b520 2026-02-22T20:11:41.751 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:41.749+0000 7f8da6485640 -1 reset not still connected to 0x55abea6dcea0 2026-02-22T20:11:41.751 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:41.749+0000 7f8da6485640 -1 reset not still connected to 0x55abf6e176c0 2026-02-22T20:11:41.751 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:41.749+0000 7f8da6485640 -1 reset not still connected to 0x55abf7185e10 2026-02-22T20:11:41.751 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:41.749+0000 7f8da6485640 -1 reset not still connected to 0x55abf96af520 2026-02-22T20:11:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 195753984 bytes 2026-02-22T20:11:42.154 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:42.153+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:11:42.154 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:42.153+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:11:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 238746624 bytes 2026-02-22T20:11:43.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 262924288 bytes 2026-02-22T20:11:43.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 300584960 bytes 2026-02-22T20:11:44.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 351674368 bytes 2026-02-22T20:11:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 437400576 bytes 2026-02-22T20:11:47.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 483861504 bytes 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_32 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 journal 128M 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_32 128M 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:47.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_32 2026-02-22T20:11:47.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:11:47.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:47.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:11:47.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:11:47.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:11:47.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:11:47.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_32 2026-02-22T20:11:47.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:47.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:47.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:11:47.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:11:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=6 2026-02-22T20:11:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:11:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:11:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 638976 bytes 2026-02-22T20:11:48.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 70114304 bytes 2026-02-22T20:11:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 82942976 bytes 2026-02-22T20:11:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 104245248 bytes 2026-02-22T20:11:50.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:11:50.672+0000 7f98c9376640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x559d9e8d03c0 finish: resending after timeout 2026-02-22T20:11:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 143875072 bytes 2026-02-22T20:11:51.649 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:51.645+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:11:51.650 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:51.645+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:11:51.650 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:51.645+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:11:51.650 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:51.645+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bff4dd0 2026-02-22T20:11:51.653 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:51.649+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:11:51.653 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:51.649+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:11:51.653 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:51.649+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:11:51.653 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:11:51.649+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:11:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 193575936 bytes 2026-02-22T20:11:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 219225088 bytes 2026-02-22T20:11:53.156 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:53.153+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:53.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 230146048 bytes 2026-02-22T20:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_33 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 journal 128M 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_33 128M 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:11:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:53.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:11:53.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:11:53.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:11:53.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:53.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_33 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_33 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:11:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:11:53.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T20:11:53.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:11:53.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:11:53.551 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:53.549+0000 7f65e934d640 -1 reset not still connected to 0x55f6db9569c0 2026-02-22T20:11:53.551 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:53.549+0000 7f65e934d640 -1 reset not still connected to 0x55f6df896680 2026-02-22T20:11:53.551 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:11:53.549+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7beeea0 2026-02-22T20:11:54.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 53601280 bytes 2026-02-22T20:11:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 84698112 bytes 2026-02-22T20:11:56.141 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.137+0000 7f8da6485640 -1 reset not still connected to 0x55abeb48b520 2026-02-22T20:11:56.142 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.141+0000 7f8da6485640 -1 reset not still connected to 0x55abeb57d1e0 2026-02-22T20:11:56.142 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.141+0000 7f8da6485640 -1 reset not still connected to 0x55abf35e32b0 2026-02-22T20:11:56.142 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.141+0000 7f8da6485640 -1 reset not still connected to 0x55abf4df24e0 2026-02-22T20:11:56.143 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.141+0000 7f8da6485640 -1 reset not still connected to 0x55abf4e1f450 2026-02-22T20:11:56.143 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.141+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:11:56.306 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.305+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:11:56.307 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.305+0000 7f8da6485640 -1 reset not still connected to 0x55abc5af1110 2026-02-22T20:11:56.307 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.305+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:11:56.307 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.305+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:11:56.307 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.305+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:11:56.307 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:56.305+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:11:56.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 136030208 bytes 2026-02-22T20:11:57.725 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:57.721+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:11:57.726 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:57.725+0000 7f8da6485640 -1 reset not still connected to 0x55abc5af1110 2026-02-22T20:11:57.727 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:57.725+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:11:57.727 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:57.725+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:11:57.727 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:57.725+0000 7f8da6485640 -1 reset not still connected to 0x55abea5425b0 2026-02-22T20:11:57.727 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:11:57.725+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_34 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 journal 128M 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:11:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_34 128M 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:11:58.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_34 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_34 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:11:58.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:11:58.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:11:58.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=7 2026-02-22T20:11:58.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:11:58.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:11:59.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 36204544 bytes 2026-02-22T20:11:59.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 64817152 bytes 2026-02-22T20:12:01.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 140919808 bytes 2026-02-22T20:12:01.899 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:01.897+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4ba59c70 2026-02-22T20:12:01.899 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:01.897+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4c358000 2026-02-22T20:12:01.899 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:01.897+0000 7f5e092a6640 -1 reset not still connected to 0x55ac519be1a0 2026-02-22T20:12:01.899 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:01.897+0000 7f5e092a6640 -1 reset not still connected to 0x55ac51ffdd40 2026-02-22T20:12:01.899 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:01.897+0000 7f5e092a6640 -1 reset not still connected to 0x55ac520c1ad0 2026-02-22T20:12:02.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 205521920 bytes 2026-02-22T20:12:03.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 236820480 bytes 2026-02-22T20:12:05.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:12:05.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:12:05.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:12:05.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:12:05.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_35 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 journal 128M 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_35 128M 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_35 2026-02-22T20:12:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:12:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:12:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:12:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:12:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:12:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_35 2026-02-22T20:12:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:05.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:05.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:12:05.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:12:05.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=10 2026-02-22T20:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:12:05.523 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:05.521+0000 7f5e092a6640 -1 reset not still connected to 0x55ac519be1a0 2026-02-22T20:12:05.523 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:05.521+0000 7f5e092a6640 -1 reset not still connected to 0x55ac51ffdd40 2026-02-22T20:12:05.523 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:05.521+0000 7f5e092a6640 -1 reset not still connected to 0x55ac520c1ad0 2026-02-22T20:12:06.445 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.441+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:12:06.445 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.441+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:12:06.446 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.445+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:12:06.446 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.445+0000 7f65e934d640 -1 reset not still connected to 0x55f6c291dba0 2026-02-22T20:12:06.446 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.445+0000 7f65e934d640 -1 reset not still connected to 0x55f6db9569c0 2026-02-22T20:12:06.447 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.445+0000 7f65e934d640 -1 reset not still connected to 0x55f6df896680 2026-02-22T20:12:06.447 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.445+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7beeea0 2026-02-22T20:12:06.447 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.445+0000 7f65e934d640 -1 reset not still connected to 0x55f6e8dd1450 2026-02-22T20:12:06.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 41891840 bytes 2026-02-22T20:12:06.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 42841088 bytes 2026-02-22T20:12:06.616 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.613+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:12:06.616 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:06.613+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:12:07.002 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.001+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0391ee0 2026-02-22T20:12:07.002 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.001+0000 7f65e934d640 -1 reset not still connected to 0x55f6e07a0270 2026-02-22T20:12:07.002 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.001+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:12:07.002 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.001+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:12:07.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 90474496 bytes 2026-02-22T20:12:07.839 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.837+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:12:07.840 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.837+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:12:07.841 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.837+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:12:07.841 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.837+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:12:07.841 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.837+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:12:07.841 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.837+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:12:07.841 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:07.837+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:12:08.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 121829376 bytes 2026-02-22T20:12:08.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 123566080 bytes 2026-02-22T20:12:08.804 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:08.800+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:12:08.805 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:08.800+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:12:09.400 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:09.396+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:12:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 186637312 bytes 2026-02-22T20:12:09.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 206319616 bytes 2026-02-22T20:12:10.323 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:10.320+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:12:10.323 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:10.320+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:12:10.324 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:10.320+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:12:11.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 256652288 bytes 2026-02-22T20:12:11.307 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:11.304+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:12:11.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 297377792 bytes 2026-02-22T20:12:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 346996736 bytes 2026-02-22T20:12:13.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 391864320 bytes 2026-02-22T20:12:13.882 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:13.880+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:12:14.038 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:14.036+0000 7f5e092a6640 -1 reset not still connected to 0x55ac51ffdd40 2026-02-22T20:12:14.038 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:14.036+0000 7f5e092a6640 -1 reset not still connected to 0x55ac520c1ad0 2026-02-22T20:12:14.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 445572096 bytes 2026-02-22T20:12:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:12:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_36 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 journal 128M 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:12:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_36 128M 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:12:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:12:15.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:15.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_36 2026-02-22T20:12:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:12:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:12:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:12:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:12:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:12:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_36 2026-02-22T20:12:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:12:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:12:15.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T20:12:15.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:12:15.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:12:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 62414848 bytes 2026-02-22T20:12:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 90544128 bytes 2026-02-22T20:12:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 151242752 bytes 2026-02-22T20:12:18.901 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:18.896+0000 7f5e092a6640 -1 reset not still connected to 0x55ac51ffdd40 2026-02-22T20:12:18.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 174035968 bytes 2026-02-22T20:12:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 250028032 bytes 2026-02-22T20:12:20.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 288587776 bytes 2026-02-22T20:12:20.622 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:20.620+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:12:20.627 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:20.624+0000 7f65e934d640 -1 reset not still connected to 0x55f6d6219040 2026-02-22T20:12:20.627 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:20.624+0000 7f65e934d640 -1 reset not still connected to 0x55f6dde53520 2026-02-22T20:12:20.627 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:20.624+0000 7f65e934d640 -1 reset not still connected to 0x55f6de693860 2026-02-22T20:12:20.627 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:20.624+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0391ee0 2026-02-22T20:12:20.628 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:20.624+0000 7f65e934d640 -1 reset not still connected to 0x55f6e07a0270 2026-02-22T20:12:20.628 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:20.624+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:12:20.628 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:20.624+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:12:21.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 320871424 bytes 2026-02-22T20:12:21.522 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:21.520+0000 7f8da6485640 -1 reset not still connected to 0x55abf6e176c0 2026-02-22T20:12:21.522 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:21.520+0000 7f8da6485640 -1 reset not still connected to 0x55abf96af520 2026-02-22T20:12:21.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 346717184 bytes 2026-02-22T20:12:22.170 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:22.168+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:12:22.171 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:22.168+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:12:22.171 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:22.168+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:12:22.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 366322688 bytes 2026-02-22T20:12:22.696 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:22.692+0000 7f8da6485640 -1 reset not still connected to 0x55abeab936c0 2026-02-22T20:12:22.696 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:22.692+0000 7f8da6485640 -1 reset not still connected to 0x55abeafd8820 2026-02-22T20:12:22.696 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:22.692+0000 7f8da6485640 -1 reset not still connected to 0x55abeb48b520 2026-02-22T20:12:22.696 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:22.692+0000 7f8da6485640 -1 reset not still connected to 0x55abeb57d1e0 2026-02-22T20:12:22.696 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:22.692+0000 7f8da6485640 -1 reset not still connected to 0x55abf35e32b0 2026-02-22T20:12:22.696 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:22.692+0000 7f8da6485640 -1 reset not still connected to 0x55abf4df24e0 2026-02-22T20:12:22.696 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:22.692+0000 7f8da6485640 -1 reset not still connected to 0x55abf4e1f450 2026-02-22T20:12:22.696 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:22.692+0000 7f8da6485640 -1 reset not still connected to 0x55abf6c2ac30 2026-02-22T20:12:22.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 393072640 bytes 2026-02-22T20:12:23.647 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:23.644+0000 7f8da6485640 -1 reset not still connected to 0x55abd9d22000 2026-02-22T20:12:23.647 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:23.644+0000 7f8da6485640 -1 reset not still connected to 0x55abf96e7040 2026-02-22T20:12:23.648 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:23.644+0000 7f8da6485640 -1 reset not still connected to 0x55abf9739860 2026-02-22T20:12:23.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 481484800 bytes 2026-02-22T20:12:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 530069504 bytes 2026-02-22T20:12:24.856 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:24.852+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3ea69ba0 2026-02-22T20:12:24.857 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:24.852+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc1110 2026-02-22T20:12:24.857 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:24.852+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4b3e1110 2026-02-22T20:12:24.857 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:24.852+0000 7f5e092a6640 -1 reset not still connected to 0x55ac515aba00 2026-02-22T20:12:24.857 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:24.852+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:12:24.857 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:24.852+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:12:25.012 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:25.008+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:12:25.045 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:25.040+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3ea69ba0 2026-02-22T20:12:25.045 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:25.040+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc1110 2026-02-22T20:12:25.045 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:25.040+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4b3e1110 2026-02-22T20:12:25.046 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:25.040+0000 7f5e092a6640 -1 reset not still connected to 0x55ac515aba00 2026-02-22T20:12:25.046 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:25.044+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:12:25.046 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:25.044+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:12:25.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 602575872 bytes 2026-02-22T20:12:26.735 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2026-02-22T20:12:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2026-02-22T20:12:26.736 INFO:tasks.rbd_mirror_thrash:waiting for 0.3078412912958328 secs before reviving daemons 2026-02-22T20:12:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:12:26.724+0000 7fae57837640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 86003) UID: 1000 2026-02-22T20:12:27.044 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2026-02-22T20:12:27.044 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:12:27.143 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2026-02-22T20:12:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-22T20:12:27.143 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-22T20:12:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-22T20:12:27.190 INFO:tasks.rbd_mirror_thrash:waiting for 77.70057506264855 secs before thrashing 2026-02-22T20:12:27.217 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:27.212+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:12:27.218 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:27.216+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:12:27.219 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:27.216+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:12:27.219 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:27.216+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:12:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:12:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:12:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:12:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_37 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 journal 128M 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_37 128M 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_37 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_37 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:12:27.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:12:27.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=10 2026-02-22T20:12:27.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:12:27.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:12:27.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 622592 bytes 2026-02-22T20:12:29.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 64148480 bytes 2026-02-22T20:12:29.931 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:29.928+0000 7f8da6485640 -1 reset not still connected to 0x55abf7185e10 2026-02-22T20:12:29.931 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:29.928+0000 7f8da6485640 -1 reset not still connected to 0x55abf96af520 2026-02-22T20:12:30.035 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:30.032+0000 7f8da6485640 -1 reset not still connected to 0x55abf9739860 2026-02-22T20:12:30.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 114066432 bytes 2026-02-22T20:12:31.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 134253568 bytes 2026-02-22T20:12:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 164946944 bytes 2026-02-22T20:12:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 179192832 bytes 2026-02-22T20:12:33.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 226755584 bytes 2026-02-22T20:12:35.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 307051520 bytes 2026-02-22T20:12:36.293 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:36.288+0000 7f8da6485640 -1 reset not still connected to 0x55abc5d21c70 2026-02-22T20:12:36.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 384606208 bytes 2026-02-22T20:12:37.624 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:12:37.625 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:12:37.625 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac30cf7d40 2026-02-22T20:12:37.625 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:12:37.626 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3bb9c8f0 2026-02-22T20:12:37.626 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:12:37.626 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac539bfc70 2026-02-22T20:12:37.626 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:12:37.626 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:12:37.626 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:37.620+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_38 2026-02-22T20:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 journal 128M 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_38 128M 2026-02-22T20:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_38 2026-02-22T20:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_38 2026-02-22T20:12:37.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:37.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:37.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:12:37.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:12:37.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T20:12:37.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:12:37.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:12:38.596 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:38.592+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:12:38.596 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:38.592+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:12:38.597 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:38.592+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:12:38.598 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:38.592+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:12:38.598 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:38.592+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:12:38.598 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:38.596+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:12:38.598 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:38.596+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:12:38.599 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:38.596+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:12:39.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 47838208 bytes 2026-02-22T20:12:40.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 72992768 bytes 2026-02-22T20:12:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 140451840 bytes 2026-02-22T20:12:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:12:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:12:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:12:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:12:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:12:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_39 2026-02-22T20:12:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 journal 128M 2026-02-22T20:12:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_39 128M 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:42.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:12:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:12:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:12:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:12:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_39 2026-02-22T20:12:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:12:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:12:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:12:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:12:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:12:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_39 2026-02-22T20:12:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:42.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:42.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:12:42.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:12:42.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=10 2026-02-22T20:12:42.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:12:42.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:12:43.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 59393024 bytes 2026-02-22T20:12:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 77190144 bytes 2026-02-22T20:12:45.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 140127232 bytes 2026-02-22T20:12:46.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 149094400 bytes 2026-02-22T20:12:47.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 174178304 bytes 2026-02-22T20:12:47.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 204474368 bytes 2026-02-22T20:12:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 264161280 bytes 2026-02-22T20:12:49.213 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:49.208+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:12:50.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 322029568 bytes 2026-02-22T20:12:50.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 367801344 bytes 2026-02-22T20:12:51.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 419295232 bytes 2026-02-22T20:12:52.192 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.188+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:12:52.193 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.188+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:12:52.193 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.188+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:12:52.195 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.192+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:12:52.195 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.192+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:12:52.195 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.192+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:12:52.195 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.192+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:12:52.195 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.192+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:12:52.196 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.192+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:12:52.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:12:52.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:12:52.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:12:52.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:12:52.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:12:52.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_40 2026-02-22T20:12:52.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 journal 128M 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_40 128M 2026-02-22T20:12:52.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_40 2026-02-22T20:12:52.764 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:52.756+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_40 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:52.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:12:52.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:12:52.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T20:12:52.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:12:52.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:12:53.523 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:53.520+0000 7f65e934d640 -1 reset not still connected to 0x55f6bf6ed450 2026-02-22T20:12:53.523 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:53.520+0000 7f65e934d640 -1 reset not still connected to 0x55f6c12dcb60 2026-02-22T20:12:53.524 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:53.520+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:12:53.525 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:53.520+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:12:53.525 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:53.520+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:12:53.525 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:53.520+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:12:53.525 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:53.520+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:12:53.526 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:53.520+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:12:54.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 69428224 bytes 2026-02-22T20:12:54.355 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.42640779831160325) > weight (0.25) 2026-02-22T20:12:54.356 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2026-02-22T20:12:54.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2026-02-22T20:12:54.356 INFO:tasks.rbd_mirror_thrash:waiting for 6.035898281710745 secs before reviving daemons 2026-02-22T20:12:54.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:12:54.345+0000 7f840ef2b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 20080) UID: 1000 2026-02-22T20:12:54.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 101559296 bytes 2026-02-22T20:12:54.954 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:54.951+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:12:54.954 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:54.951+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:12:55.543 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:55.539+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:12:55.544 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:55.539+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:12:55.544 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:55.539+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:12:55.545 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:55.539+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:12:55.545 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:55.539+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:12:55.545 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:55.539+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:12:55.545 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:55.539+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:12:55.545 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:55.539+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:12:55.545 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:55.539+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:12:55.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 120725504 bytes 2026-02-22T20:12:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 139025408 bytes 2026-02-22T20:12:56.222 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:56.219+0000 7f8da6485640 -1 reset not still connected to 0x55abe23bd6c0 2026-02-22T20:12:56.222 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:56.219+0000 7f8da6485640 -1 reset not still connected to 0x55abea6dcea0 2026-02-22T20:12:56.222 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:56.219+0000 7f8da6485640 -1 reset not still connected to 0x55abf6e176c0 2026-02-22T20:12:56.222 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:56.219+0000 7f8da6485640 -1 reset not still connected to 0x55abf7185e10 2026-02-22T20:12:56.222 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:56.219+0000 7f8da6485640 -1 reset not still connected to 0x55abf96af520 2026-02-22T20:12:56.222 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:56.219+0000 7f8da6485640 -1 reset not still connected to 0x55abd433cb60 2026-02-22T20:12:56.222 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:56.219+0000 7f8da6485640 -1 reset not still connected to 0x55abd74e2dd0 2026-02-22T20:12:56.222 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:12:56.219+0000 7f8da6485640 -1 reset not still connected to 0x55abddd5e5b0 2026-02-22T20:12:56.382 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:56.379+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2fb432b0 2026-02-22T20:12:56.383 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:56.379+0000 7f5e092a6640 -1 reset not still connected to 0x55ac342ab520 2026-02-22T20:12:56.384 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:56.379+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3dc3cd00 2026-02-22T20:12:56.384 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:56.379+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3de05450 2026-02-22T20:12:56.384 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:56.379+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4df58c30 2026-02-22T20:12:56.384 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:56.379+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:12:56.384 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:56.379+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:12:56.385 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:56.379+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:12:57.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:12:57.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:12:57.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:12:57.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_41 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 journal 128M 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_41 128M 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:12:57.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:12:57.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_41 2026-02-22T20:12:57.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:12:57.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:57.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:12:57.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:12:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:12:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:12:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_41 2026-02-22T20:12:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:12:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:12:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:12:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:12:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=14 2026-02-22T20:12:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:12:57.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:12:58.143 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:58.139+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:12:58.144 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:12:58.139+0000 7f65e934d640 -1 reset not still connected to 0x55f6f832f860 2026-02-22T20:12:59.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 68239360 bytes 2026-02-22T20:12:59.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 68858880 bytes 2026-02-22T20:12:59.739 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:12:59.735+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:12:59.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:12:59.905+0000 7f98c9b77640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x559db4549b00 finish: resending after timeout 2026-02-22T20:13:00.393 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2026-02-22T20:13:00.394 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:13:00.394 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2026-02-22T20:13:00.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-22T20:13:00.394 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-22T20:13:00.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-22T20:13:00.397 INFO:tasks.rbd_mirror_thrash:waiting for 101.17796231513427 secs before thrashing 2026-02-22T20:13:00.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 108057600 bytes 2026-02-22T20:13:01.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 134551552 bytes 2026-02-22T20:13:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 151009280 bytes 2026-02-22T20:13:02.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 172260352 bytes 2026-02-22T20:13:03.077 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:03.075+0000 7f8da6485640 -1 reset not still connected to 0x55abf9739860 2026-02-22T20:13:03.272 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:03.271+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:13:03.272 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:03.271+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:13:04.301 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:04.299+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:13:04.304 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:04.303+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:13:04.304 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:04.303+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:13:04.304 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:04.303+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:13:04.668 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:04.667+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:13:04.668 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:04.667+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:13:04.668 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:04.667+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:13:04.668 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:04.667+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:13:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 256732160 bytes 2026-02-22T20:13:05.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 278818816 bytes 2026-02-22T20:13:05.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 304311296 bytes 2026-02-22T20:13:05.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:13:05.661+0000 7f98c9376640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x559db4549b00 finish: resending after timeout 2026-02-22T20:13:06.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 371481600 bytes 2026-02-22T20:13:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 406003712 bytes 2026-02-22T20:13:07.142 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:07.143+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:13:07.143 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:07.143+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:13:07.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 444037120 bytes 2026-02-22T20:13:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 499058688 bytes 2026-02-22T20:13:08.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 499683328 bytes 2026-02-22T20:13:09.146 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:09.147+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3332c0d0 2026-02-22T20:13:09.147 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:09.147+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3ea69ba0 2026-02-22T20:13:09.147 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:09.147+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc1110 2026-02-22T20:13:09.147 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:09.147+0000 7f5e092a6640 -1 reset not still connected to 0x55ac515aba00 2026-02-22T20:13:09.147 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:09.147+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:13:09.147 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:09.147+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:13:10.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 563954688 bytes 2026-02-22T20:13:11.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 625536000 bytes 2026-02-22T20:13:11.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:13:11.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:13:11.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:13:11.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:13:11.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:13:11.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_42 2026-02-22T20:13:11.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 journal 128M 2026-02-22T20:13:11.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_42 128M 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:11.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:11.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:11.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:13:11.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:11.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:13:11.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:13:11.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:13:11.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:11.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_42 2026-02-22T20:13:11.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:13:11.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_42 2026-02-22T20:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=7 2026-02-22T20:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:13:11.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:13:13.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 56313856 bytes 2026-02-22T20:13:13.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 80877568 bytes 2026-02-22T20:13:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 104548352 bytes 2026-02-22T20:13:15.300 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.299+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:13:15.300 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.299+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:13:15.301 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.299+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:13:15.302 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.299+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:13:15.302 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.299+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:13:15.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 136823808 bytes 2026-02-22T20:13:15.495 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:13:15.496 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:13:15.496 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:13:15.497 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:13:15.497 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:13:15.497 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:13:15.497 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:13:15.498 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:13:15.498 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:13:15.498 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6f832f860 2026-02-22T20:13:15.498 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:15.495+0000 7f65e934d640 -1 reset not still connected to 0x55f6fb4441a0 2026-02-22T20:13:15.784 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:15.783+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:13:15.785 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:15.783+0000 7f8da6485640 -1 reset not still connected to 0x55abc5af1110 2026-02-22T20:13:15.785 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:15.783+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:13:15.785 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:15.783+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:13:15.787 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:15.787+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:13:15.787 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:15.787+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:13:16.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 181140480 bytes 2026-02-22T20:13:17.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 226429952 bytes 2026-02-22T20:13:17.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 241490944 bytes 2026-02-22T20:13:18.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 300731392 bytes 2026-02-22T20:13:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:13:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:13:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:13:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:13:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:13:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_43 2026-02-22T20:13:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 journal 128M 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_43 128M 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:13:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:13:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:13:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:13:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_43 2026-02-22T20:13:18.986 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:18.987+0000 7f65e934d640 -1 reset not still connected to 0x55f6dde53520 2026-02-22T20:13:18.986 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:18.987+0000 7f65e934d640 -1 reset not still connected to 0x55f6de693860 2026-02-22T20:13:18.986 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:18.987+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0391ee0 2026-02-22T20:13:18.987 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:18.987+0000 7f65e934d640 -1 reset not still connected to 0x55f6e07a0270 2026-02-22T20:13:18.987 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:18.987+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:13:18.987 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:18.987+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:13:18.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_43 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:13:18.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:13:18.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=7 2026-02-22T20:13:18.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:13:18.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:13:19.080 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:19.079+0000 7f65e934d640 -1 reset not still connected to 0x55f6c12dcb60 2026-02-22T20:13:19.080 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:19.079+0000 7f65e934d640 -1 reset not still connected to 0x55f6c245b1e0 2026-02-22T20:13:19.080 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:19.079+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:13:19.081 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:19.079+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:13:19.081 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:19.079+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:13:19.081 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:19.079+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:13:19.081 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:19.079+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:13:19.081 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:19.079+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:13:19.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 15636480 bytes 2026-02-22T20:13:19.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 31692800 bytes 2026-02-22T20:13:20.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 82424832 bytes 2026-02-22T20:13:22.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 128260096 bytes 2026-02-22T20:13:22.280 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:13:22.265+0000 7f557ffc8640 -1 reset not still connected to 0x55922b795860 2026-02-22T20:13:22.280 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:13:22.265+0000 7f557ffc8640 -1 reset not still connected to 0x559236aac1a0 2026-02-22T20:13:22.280 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:13:22.265+0000 7f557ffc8640 -1 reset not still connected to 0x55923ab084e0 2026-02-22T20:13:22.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:13:22.285+0000 7f98c9b77640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x559db4549b00 finish: resending after timeout 2026-02-22T20:13:23.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 265083904 bytes 2026-02-22T20:13:23.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 270971904 bytes 2026-02-22T20:13:24.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 341177344 bytes 2026-02-22T20:13:25.824 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:25.823+0000 7f65e934d640 -1 reset not still connected to 0x55f6e1851450 2026-02-22T20:13:25.825 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:25.827+0000 7f65e934d640 -1 reset not still connected to 0x55f6e2a605b0 2026-02-22T20:13:25.826 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:25.827+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:13:25.826 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:25.827+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:13:26.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_44 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 journal 128M 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:26.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_44 128M 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:13:26.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:13:26.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:13:26.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:26.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_44 2026-02-22T20:13:26.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_44 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:13:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:13:26.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=5 2026-02-22T20:13:26.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:13:26.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:13:26.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 33489920 bytes 2026-02-22T20:13:27.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:13:27.285+0000 7f98c9b77640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x559db4549b00 finish: resending after timeout 2026-02-22T20:13:27.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 94416896 bytes 2026-02-22T20:13:29.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 151317504 bytes 2026-02-22T20:13:29.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 200220672 bytes 2026-02-22T20:13:30.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 223741952 bytes 2026-02-22T20:13:30.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 224353280 bytes 2026-02-22T20:13:30.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 240243712 bytes 2026-02-22T20:13:31.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:13:31.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:13:31.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:13:31.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:13:31.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_45 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 journal 128M 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:13:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_45 128M 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_45 2026-02-22T20:13:31.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:13:31.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:13:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:13:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:13:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:13:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_45 2026-02-22T20:13:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:13:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:13:31.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=13 2026-02-22T20:13:31.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:13:31.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:13:31.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 42733568 bytes 2026-02-22T20:13:32.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 43345920 bytes 2026-02-22T20:13:32.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 73559040 bytes 2026-02-22T20:13:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 84269056 bytes 2026-02-22T20:13:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 141369344 bytes 2026-02-22T20:13:35.404 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:35.403+0000 7f8da6485640 -1 reset not still connected to 0x55abddd5b790 2026-02-22T20:13:35.405 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:35.403+0000 7f8da6485640 -1 reset not still connected to 0x55abddd5e5b0 2026-02-22T20:13:35.405 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:35.403+0000 7f8da6485640 -1 reset not still connected to 0x55abe5309e10 2026-02-22T20:13:35.405 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:35.403+0000 7f8da6485640 -1 reset not still connected to 0x55abf552b860 2026-02-22T20:13:36.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 205652992 bytes 2026-02-22T20:13:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 209780736 bytes 2026-02-22T20:13:36.472 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T20:13:36.457+0000 7fdfce879640 -1 reset not still connected to 0x555a72ff9790 2026-02-22T20:13:36.472 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T20:13:36.457+0000 7fdfce879640 -1 reset not still connected to 0x555a8e966ea0 2026-02-22T20:13:36.472 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T20:13:36.457+0000 7fdfce879640 -1 reset not still connected to 0x555aa006d790 2026-02-22T20:13:36.686 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:36.687+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:13:36.686 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:36.687+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:13:36.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 245243904 bytes 2026-02-22T20:13:37.070 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:37.071+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:13:37.071 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:37.071+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:13:37.842 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:37.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:13:37.843 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:37.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:13:37.843 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:37.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:13:37.843 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:37.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:13:37.843 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:37.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:13:38.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 312779776 bytes 2026-02-22T20:13:38.844 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:38.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:13:38.844 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:38.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:13:38.844 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:38.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:13:38.845 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:38.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:13:38.845 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:38.843+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:13:38.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 373587968 bytes 2026-02-22T20:13:39.375 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:39.375+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:13:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 402283520 bytes 2026-02-22T20:13:40.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 466583552 bytes 2026-02-22T20:13:41.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 534419456 bytes 2026-02-22T20:13:42.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 578967552 bytes 2026-02-22T20:13:43.116 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:43.114+0000 7f65e934d640 -1 reset not still connected to 0x55f6e1851450 2026-02-22T20:13:43.117 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:43.114+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:13:43.117 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:43.114+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:13:43.432 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.430+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2fb432b0 2026-02-22T20:13:43.432 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.430+0000 7f5e092a6640 -1 reset not still connected to 0x55ac342ab520 2026-02-22T20:13:43.436 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3dc3cd00 2026-02-22T20:13:43.436 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3de05450 2026-02-22T20:13:43.436 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac415984e0 2026-02-22T20:13:43.436 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac415cc5b0 2026-02-22T20:13:43.436 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4df58c30 2026-02-22T20:13:43.436 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:13:43.437 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:13:43.437 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:13:43.437 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:13:43.438 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2fb432b0 2026-02-22T20:13:43.438 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac342ab520 2026-02-22T20:13:43.438 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3dc3cd00 2026-02-22T20:13:43.438 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:13:43.438 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:13:43.438 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.434+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:13:43.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 639321088 bytes 2026-02-22T20:13:43.804 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.802+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:13:43.804 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.802+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:13:43.804 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:43.802+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:13:44.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:13:44.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_46 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 journal 128M 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:44.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_46 128M 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:13:44.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:44.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:13:44.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:13:44.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:13:44.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:44.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_46 2026-02-22T20:13:44.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:13:44.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:44.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:13:44.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:13:44.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:13:44.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:13:44.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_46 2026-02-22T20:13:44.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:44.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:44.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:13:44.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:13:44.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=9 2026-02-22T20:13:44.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:13:44.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:13:44.891 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6540967670219465) > weight (0.25) 2026-02-22T20:13:44.892 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.46193431956375164) > weight (0.25) 2026-02-22T20:13:44.892 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5378297477100537) > weight (0.25) 2026-02-22T20:13:44.892 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5461072334937805) > weight (0.25) 2026-02-22T20:13:44.892 INFO:tasks.rbd_mirror_thrash:waiting for 71.64397788178229 secs before thrashing 2026-02-22T20:13:45.278 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.278+0000 7f65e934d640 -1 reset not still connected to 0x55f6da168820 2026-02-22T20:13:45.278 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.278+0000 7f65e934d640 -1 reset not still connected to 0x55f6e1851450 2026-02-22T20:13:45.279 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.278+0000 7f65e934d640 -1 reset not still connected to 0x55f6e2a605b0 2026-02-22T20:13:45.280 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.278+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:13:45.280 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.278+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:13:45.297 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.294+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:13:45.297 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.294+0000 7f65e934d640 -1 reset not still connected to 0x55f6cadef450 2026-02-22T20:13:45.298 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:13:45.298 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:13:45.299 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:13:45.300 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:13:45.300 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:13:45.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 77385728 bytes 2026-02-22T20:13:45.814 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.810+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:13:45.814 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.810+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:13:45.815 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.814+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:13:45.815 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.814+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:13:45.815 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.814+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:13:45.815 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:45.814+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:13:45.935 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:45.934+0000 7f8da6485640 -1 reset not still connected to 0x55abf9739860 2026-02-22T20:13:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 110632960 bytes 2026-02-22T20:13:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 113313792 bytes 2026-02-22T20:13:46.811 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:46.810+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:13:46.811 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:46.810+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc20d0 2026-02-22T20:13:46.812 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:46.810+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:13:46.812 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:46.810+0000 7f5e092a6640 -1 reset not still connected to 0x55ac539bfc70 2026-02-22T20:13:46.812 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:46.810+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:13:46.813 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:46.810+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:13:46.813 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:46.810+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:13:46.853 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:46.850+0000 7f65e934d640 -1 reset not still connected to 0x55f6e1851450 2026-02-22T20:13:46.854 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:46.854+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:13:46.854 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:46.854+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:13:46.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 118456320 bytes 2026-02-22T20:13:48.245 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.242+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:13:48.246 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:13:48.246 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:13:48.246 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:13:48.247 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:13:48.247 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:13:48.247 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:13:48.247 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6f832f860 2026-02-22T20:13:48.247 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6fb4441a0 2026-02-22T20:13:48.249 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:13:48.250 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.250+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:13:48.250 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.250+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:13:48.251 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.250+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:13:48.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 211136512 bytes 2026-02-22T20:13:48.863 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.862+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:13:48.863 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.862+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:13:48.863 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.862+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:13:48.863 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.862+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:13:48.863 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.862+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:13:48.863 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.862+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:13:48.864 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.862+0000 7f65e934d640 -1 reset not still connected to 0x55f6f832f860 2026-02-22T20:13:48.864 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:48.862+0000 7f65e934d640 -1 reset not still connected to 0x55f6fb4441a0 2026-02-22T20:13:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 281127936 bytes 2026-02-22T20:13:50.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 301698048 bytes 2026-02-22T20:13:51.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 344110080 bytes 2026-02-22T20:13:51.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 366272512 bytes 2026-02-22T20:13:52.244 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:52.242+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:13:52.244 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:13:52.242+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:13:52.739 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:52.738+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:13:52.740 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:52.738+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:13:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 499441664 bytes 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_47 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 journal 128M 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_47 128M 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:53.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:53.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:13:53.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:53.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:13:53.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:13:53.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:13:53.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:13:53.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_47 2026-02-22T20:13:53.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:13:53.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:53.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:13:53.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:13:53.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:13:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:13:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_47 2026-02-22T20:13:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:13:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:13:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:13:53.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:13:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=14 2026-02-22T20:13:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:13:53.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:13:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 19530752 bytes 2026-02-22T20:13:53.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 20125696 bytes 2026-02-22T20:13:54.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 32816128 bytes 2026-02-22T20:13:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 109026304 bytes 2026-02-22T20:13:55.745 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2fb432b0 2026-02-22T20:13:55.745 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac342ab520 2026-02-22T20:13:55.745 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3dc3cd00 2026-02-22T20:13:55.745 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3de05450 2026-02-22T20:13:55.745 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac415cc5b0 2026-02-22T20:13:55.746 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4df58c30 2026-02-22T20:13:55.746 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:13:55.746 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:13:55.746 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:13:55.746 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:55.742+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:13:56.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 192630784 bytes 2026-02-22T20:13:56.188 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:56.186+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:13:56.497 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.494+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:13:56.497 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.494+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:13:56.498 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.498+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:13:56.498 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.498+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:13:56.498 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.498+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:13:56.498 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.498+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:13:56.499 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.498+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:13:56.499 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.498+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:13:56.499 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.498+0000 7f65e934d640 -1 reset not still connected to 0x55f6f832f860 2026-02-22T20:13:56.499 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.498+0000 7f65e934d640 -1 reset not still connected to 0x55f6fb4441a0 2026-02-22T20:13:56.952 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.950+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:13:56.952 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:56.950+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:13:57.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 242565120 bytes 2026-02-22T20:13:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 244722688 bytes 2026-02-22T20:13:57.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 275061760 bytes 2026-02-22T20:13:58.680 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:58.678+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2456d00 2026-02-22T20:13:58.680 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:58.678+0000 7f65e934d640 -1 reset not still connected to 0x55f6d936f860 2026-02-22T20:13:58.681 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:58.682+0000 7f65e934d640 -1 reset not still connected to 0x55f6da168820 2026-02-22T20:13:58.682 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:58.682+0000 7f65e934d640 -1 reset not still connected to 0x55f6e1851450 2026-02-22T20:13:58.682 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:58.682+0000 7f65e934d640 -1 reset not still connected to 0x55f6e2a605b0 2026-02-22T20:13:58.682 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:58.682+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:13:58.682 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:13:58.682+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:13:58.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 352249856 bytes 2026-02-22T20:13:59.783 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:59.782+0000 7f5e092a6640 -1 reset not still connected to 0x55ac515aba00 2026-02-22T20:13:59.784 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:59.782+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:13:59.784 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:13:59.782+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:14:00.064 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:00.062+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:14:00.065 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:00.062+0000 7f5e092a6640 -1 reset not still connected to 0x55ac30cf7d40 2026-02-22T20:14:00.065 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:00.062+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:14:00.067 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:00.066+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc20d0 2026-02-22T20:14:00.067 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:00.066+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:14:00.068 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:00.066+0000 7f5e092a6640 -1 reset not still connected to 0x55ac539bfc70 2026-02-22T20:14:00.068 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:00.066+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:14:00.068 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:00.066+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:14:00.068 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:00.066+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:14:00.954 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:00.954+0000 7f8da6485640 -1 reset not still connected to 0x55abf6e176c0 2026-02-22T20:14:00.955 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:00.954+0000 7f8da6485640 -1 reset not still connected to 0x55abf7185e10 2026-02-22T20:14:00.955 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:00.954+0000 7f8da6485640 -1 reset not still connected to 0x55abf96af520 2026-02-22T20:14:00.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 475891712 bytes 2026-02-22T20:14:01.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 508516352 bytes 2026-02-22T20:14:02.195 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:02.194+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4b880dd0 2026-02-22T20:14:02.195 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:02.194+0000 7f5e092a6640 -1 reset not still connected to 0x55ac565e0ea0 2026-02-22T20:14:02.197 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:02.194+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5880a410 2026-02-22T20:14:02.197 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:02.194+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:14:02.197 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:02.194+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:14:02.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 539357184 bytes 2026-02-22T20:14:02.725 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:02.722+0000 7f8da6485640 -1 reset not still connected to 0x55abddd5e5b0 2026-02-22T20:14:02.725 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:02.722+0000 7f8da6485640 -1 reset not still connected to 0x55abe5309e10 2026-02-22T20:14:02.725 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:02.722+0000 7f8da6485640 -1 reset not still connected to 0x55abf552b860 2026-02-22T20:14:02.849 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:02.846+0000 7f65e934d640 -1 reset not still connected to 0x55f6e1851450 2026-02-22T20:14:02.852 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:02.850+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:14:02.853 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:02.850+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:14:03.416 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:03.414+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:14:03.417 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:03.414+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:14:03.417 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:03.414+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:14:03.417 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:03.414+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:14:03.488 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:03.486+0000 7f65e934d640 -1 reset not still connected to 0x55f6bf6ed450 2026-02-22T20:14:03.489 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:03.486+0000 7f65e934d640 -1 reset not still connected to 0x55f6c245b1e0 2026-02-22T20:14:03.489 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:03.486+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:14:03.489 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:03.486+0000 7f65e934d640 -1 reset not still connected to 0x55f6c8ec9a00 2026-02-22T20:14:03.489 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:03.486+0000 7f65e934d640 -1 reset not still connected to 0x55f6cadef450 2026-02-22T20:14:03.489 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:03.486+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:14:03.489 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:03.486+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 632528896 bytes 2026-02-22T20:14:03.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 633135104 bytes 2026-02-22T20:14:04.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 653971456 bytes 2026-02-22T20:14:04.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 690643968 bytes 2026-02-22T20:14:05.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1700 IOs, 714895360 bytes 2026-02-22T20:14:05.247 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:14:05.247 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6c8ec9a00 2026-02-22T20:14:05.248 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6cadef450 2026-02-22T20:14:05.248 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:14:05.248 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:14:05.248 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:05.249 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:14:05.249 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.246+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:14:05.537 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.534+0000 7f65e934d640 -1 reset not still connected to 0x55f6da168820 2026-02-22T20:14:05.537 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.534+0000 7f65e934d640 -1 reset not still connected to 0x55f6e1851450 2026-02-22T20:14:05.538 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.538+0000 7f65e934d640 -1 reset not still connected to 0x55f6e2a605b0 2026-02-22T20:14:05.538 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.538+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:14:05.539 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.538+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:14:05.561 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.558+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:05.562 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.562+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:14:05.563 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:05.562+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:14:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1800 IOs, 768952320 bytes 2026-02-22T20:14:06.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1900 IOs, 799232000 bytes 2026-02-22T20:14:07.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:14:07.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:14:07.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:14:07.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:14:07.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:14:07.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_48 2026-02-22T20:14:07.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 journal 128M 2026-02-22T20:14:07.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_48 128M 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:14:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:07.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_48 2026-02-22T20:14:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:14:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:14:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:14:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:14:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:14:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_48 2026-02-22T20:14:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:07.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:14:07.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:14:07.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=9 2026-02-22T20:14:07.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:14:07.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:14:07.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 18809856 bytes 2026-02-22T20:14:07.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 27356160 bytes 2026-02-22T20:14:08.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 48177152 bytes 2026-02-22T20:14:08.342 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:08.342+0000 7f5e092a6640 -1 reset not still connected to 0x55ac30cf7d40 2026-02-22T20:14:08.343 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:08.342+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:14:08.343 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:08.342+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc20d0 2026-02-22T20:14:08.344 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:08.342+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:14:08.344 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:08.342+0000 7f5e092a6640 -1 reset not still connected to 0x55ac539bfc70 2026-02-22T20:14:08.344 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:08.342+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:14:08.345 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:08.342+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:14:08.345 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:08.342+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:14:08.574 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:08.574+0000 7f65e934d640 -1 reset not still connected to 0x55f6e07a0270 2026-02-22T20:14:08.574 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:08.574+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:14:08.574 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:08.574+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:14:09.235 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:09.234+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:14:09.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 106537984 bytes 2026-02-22T20:14:09.413 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:09.410+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:14:10.131 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:10.130+0000 7f5e092a6640 -1 reset not still connected to 0x55ac515aba00 2026-02-22T20:14:10.132 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:10.130+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:14:10.133 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:10.130+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:14:10.850 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:10.850+0000 7f8da6485640 -1 reset not still connected to 0x55abe5309e10 2026-02-22T20:14:10.851 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:10.850+0000 7f8da6485640 -1 reset not still connected to 0x55abf552b860 2026-02-22T20:14:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 174183424 bytes 2026-02-22T20:14:12.296 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.294+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:14:12.297 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:14:12.297 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:14:12.297 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:14:12.298 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:14:12.298 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:14:12.298 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:14:12.298 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6f832f860 2026-02-22T20:14:12.298 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6fb4441a0 2026-02-22T20:14:12.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 249249792 bytes 2026-02-22T20:14:12.888 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.886+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:14:12.888 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:12.886+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:14:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 249874432 bytes 2026-02-22T20:14:14.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 307603456 bytes 2026-02-22T20:14:14.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 308243456 bytes 2026-02-22T20:14:14.713 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.710+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:14:14.713 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.710+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:14:14.713 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.710+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:14.714 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.714+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:14:14.714 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.714+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:14:14.756 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.754+0000 7f65e934d640 -1 reset not still connected to 0x55f6d936f860 2026-02-22T20:14:14.756 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.754+0000 7f65e934d640 -1 reset not still connected to 0x55f6da168820 2026-02-22T20:14:14.756 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.754+0000 7f65e934d640 -1 reset not still connected to 0x55f6e1851450 2026-02-22T20:14:14.757 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.754+0000 7f65e934d640 -1 reset not still connected to 0x55f6e2a605b0 2026-02-22T20:14:14.757 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.754+0000 7f65e934d640 -1 reset not still connected to 0x55f6f4f5c270 2026-02-22T20:14:14.757 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.754+0000 7f65e934d640 -1 reset not still connected to 0x55f6f8320dd0 2026-02-22T20:14:14.757 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:14.754+0000 7f65e934d640 -1 reset not still connected to 0x55f6f833c270 2026-02-22T20:14:14.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 353333248 bytes 2026-02-22T20:14:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 363408384 bytes 2026-02-22T20:14:15.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 408784896 bytes 2026-02-22T20:14:16.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:14:16.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:14:16.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:14:16.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:14:16.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:14:16.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_49 2026-02-22T20:14:16.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 journal 128M 2026-02-22T20:14:16.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:16.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_49 128M 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:16.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:14:16.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:16.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:14:16.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:14:16.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:14:16.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:16.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_49 2026-02-22T20:14:16.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:14:16.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:16.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:14:16.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:14:16.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:14:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:14:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_49 2026-02-22T20:14:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:14:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:14:16.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=13 2026-02-22T20:14:16.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:14:16.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:14:17.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 42489856 bytes 2026-02-22T20:14:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 62085120 bytes 2026-02-22T20:14:18.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 91646976 bytes 2026-02-22T20:14:18.704 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:18.702+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:14:18.705 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:18.702+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:14:18.705 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:18.702+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:18.706 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:18.706+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:14:18.707 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:18.706+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:14:19.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 121510912 bytes 2026-02-22T20:14:19.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 144002048 bytes 2026-02-22T20:14:20.486 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0961a0 2026-02-22T20:14:20.487 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:14:20.487 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:14:20.487 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac30cf7d40 2026-02-22T20:14:20.487 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:14:20.488 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc20d0 2026-02-22T20:14:20.488 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:14:20.488 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac539bfc70 2026-02-22T20:14:20.488 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:14:20.488 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:14:20.489 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:20.486+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:14:20.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 202522624 bytes 2026-02-22T20:14:21.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 264235008 bytes 2026-02-22T20:14:22.482 INFO:tasks.ceph.cluster2.osd.0.trial066.stderr:2026-02-22T20:14:22.469+0000 7f21cdd98640 -1 reset not still connected to 0x5578b180f110 2026-02-22T20:14:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 333891584 bytes 2026-02-22T20:14:23.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 334485504 bytes 2026-02-22T20:14:23.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 373088256 bytes 2026-02-22T20:14:23.965 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:23.962+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:14:24.300 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:24.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:14:24.300 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:24.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:14:24.300 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:24.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:24.301 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:24.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:14:24.301 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:24.298+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:14:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 426622976 bytes 2026-02-22T20:14:25.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 456127488 bytes 2026-02-22T20:14:25.536 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:25.534+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:14:25.536 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:25.534+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:14:25.537 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:25.534+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:14:25.537 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:25.534+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:14:25.537 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:25.534+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:14:25.537 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:25.534+0000 7f65e934d640 -1 reset not still connected to 0x55f6f832f860 2026-02-22T20:14:25.537 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:25.534+0000 7f65e934d640 -1 reset not still connected to 0x55f6fb4441a0 2026-02-22T20:14:25.772 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e0961a0 2026-02-22T20:14:25.772 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:14:25.772 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:14:25.773 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac30cf7d40 2026-02-22T20:14:25.774 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:14:25.774 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc20d0 2026-02-22T20:14:25.774 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:14:25.774 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac539bfc70 2026-02-22T20:14:25.774 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:14:25.775 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:14:25.775 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:25.770+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:14:26.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 510733312 bytes 2026-02-22T20:14:26.559 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.558+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:26.570 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.570+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:14:26.570 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.570+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:26.570 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.570+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:14:26.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 524558336 bytes 2026-02-22T20:14:26.631 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.630+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:14:26.631 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.630+0000 7f65e934d640 -1 reset not still connected to 0x55f6c8ec9a00 2026-02-22T20:14:26.631 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.630+0000 7f65e934d640 -1 reset not still connected to 0x55f6cadef450 2026-02-22T20:14:26.632 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.630+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:14:26.632 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.630+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:14:26.633 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.630+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:26.633 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.630+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:14:26.633 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:26.630+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:14:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 566614016 bytes 2026-02-22T20:14:27.715 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:27.714+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:14:27.716 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:27.714+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc20d0 2026-02-22T20:14:27.718 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:27.718+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:14:27.718 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:27.718+0000 7f5e092a6640 -1 reset not still connected to 0x55ac539bfc70 2026-02-22T20:14:27.718 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:27.718+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:14:27.718 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:27.718+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:14:27.718 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:27.718+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:14:27.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 610344960 bytes 2026-02-22T20:14:28.584 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:28.581+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:14:28.584 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:28.581+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:14:29.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:14:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:14:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:14:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:14:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:14:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_50 2026-02-22T20:14:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 journal 128M 2026-02-22T20:14:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n journal ']' 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_50 128M 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:14:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:14:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 128M ']' 2026-02-22T20:14:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=128M 2026-02-22T20:14:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:14:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_50 2026-02-22T20:14:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:14:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' pool = image ']' 2026-02-22T20:14:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ case "${pool}" in 2026-02-22T20:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 1 2026-02-22T20:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n '' ']' 2026-02-22T20:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_50 2026-02-22T20:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=11 2026-02-22T20:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:14:30.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 39957504 bytes 2026-02-22T20:14:30.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 83176448 bytes 2026-02-22T20:14:31.280 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:31.277+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:14:31.732 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:31.729+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:14:31.732 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:31.729+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:14:32.215 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:14:32.215 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:14:32.215 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac30cf7d40 2026-02-22T20:14:32.215 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:14:32.216 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc20d0 2026-02-22T20:14:32.216 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:14:32.216 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac539bfc70 2026-02-22T20:14:32.216 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:14:32.217 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:14:32.217 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:32.213+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:14:34.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 194424832 bytes 2026-02-22T20:14:34.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 239044608 bytes 2026-02-22T20:14:35.017 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:35.013+0000 7f65e934d640 -1 reset not still connected to 0x55f6d6219040 2026-02-22T20:14:35.017 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:35.013+0000 7f65e934d640 -1 reset not still connected to 0x55f6dde53520 2026-02-22T20:14:35.017 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:35.013+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0391ee0 2026-02-22T20:14:35.017 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:35.013+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:14:35.017 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:35.013+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:14:35.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 290052096 bytes 2026-02-22T20:14:36.263 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:36.261+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:14:36.263 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:36.261+0000 7f8da6485640 -1 reset not still connected to 0x55abc5af1110 2026-02-22T20:14:36.264 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:36.261+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:14:36.264 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:36.261+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:14:36.264 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:36.261+0000 7f8da6485640 -1 reset not still connected to 0x55abcb21a680 2026-02-22T20:14:36.264 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:36.261+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:14:36.264 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:36.261+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:14:36.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 354819072 bytes 2026-02-22T20:14:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 388884480 bytes 2026-02-22T20:14:37.637 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:37.633+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:14:37.639 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:37.637+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:14:37.639 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:37.637+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:14:37.639 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:37.637+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:14:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 422534144 bytes 2026-02-22T20:14:38.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 493646848 bytes 2026-02-22T20:14:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 502407168 bytes 2026-02-22T20:14:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 537124864 bytes 2026-02-22T20:14:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:14:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:14:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:14:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:14:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_pool_images cluster2 mirror 50 2026-02-22T20:14:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_count=50 2026-02-22T20:14:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:14:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local count 2026-02-22T20:14:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_count=0 2026-02-22T20:14:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:14:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 40 2026-02-22T20:14:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in `seq 1 40` 2026-02-22T20:14:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 1 -ne 1 2026-02-22T20:14:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2026-02-22T20:14:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep 'images: ' 2026-02-22T20:14:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut '-d ' -f 2 2026-02-22T20:14:40.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: failed to query services: (13) Permission denied 2026-02-22T20:14:40.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ count=50 2026-02-22T20:14:40.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 50 = 50 2026-02-22T20:14:40.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:14:40.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_pool_healthy cluster2 mirror 2026-02-22T20:14:40.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:14:40.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:40.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:14:40.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state 2026-02-22T20:14:40.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 40 2026-02-22T20:14:40.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in `seq 1 40` 2026-02-22T20:14:40.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 1 -ne 1 2026-02-22T20:14:40.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2026-02-22T20:14:40.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep 'image health:' 2026-02-22T20:14:40.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut '-d ' -f 3 2026-02-22T20:14:40.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: failed to query services: (13) Permission denied 2026-02-22T20:14:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=OK 2026-02-22T20:14:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test OK = ERROR 2026-02-22T20:14:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test OK = OK 2026-02-22T20:14:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:14:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_pool_images cluster1 mirror 50 2026-02-22T20:14:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:14:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_count=50 2026-02-22T20:14:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:14:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local count 2026-02-22T20:14:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_count=0 2026-02-22T20:14:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:14:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 40 2026-02-22T20:14:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in `seq 1 40` 2026-02-22T20:14:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 1 -ne 1 2026-02-22T20:14:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-22T20:14:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep 'images: ' 2026-02-22T20:14:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut '-d ' -f 2 2026-02-22T20:14:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: failed to query services: (13) Permission denied 2026-02-22T20:14:40.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ count=50 2026-02-22T20:14:40.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 50 = 50 2026-02-22T20:14:40.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:14:40.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_pool_healthy cluster1 mirror 2026-02-22T20:14:40.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:14:40.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:14:40.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:14:40.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state 2026-02-22T20:14:40.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 40 2026-02-22T20:14:40.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in `seq 1 40` 2026-02-22T20:14:40.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 1 -ne 1 2026-02-22T20:14:40.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-22T20:14:40.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep 'image health:' 2026-02-22T20:14:40.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut '-d ' -f 3 2026-02-22T20:14:40.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: failed to query services: (13) Permission denied 2026-02-22T20:14:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=WARNING 2026-02-22T20:14:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test WARNING = ERROR 2026-02-22T20:14:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test WARNING = OK 2026-02-22T20:14:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in `seq 1 40` 2026-02-22T20:14:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 -ne 1 2026-02-22T20:14:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 30 2026-02-22T20:14:41.575 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9075032776066075) > weight (0.25) 2026-02-22T20:14:41.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.41765750390703105) > weight (0.25) 2026-02-22T20:14:41.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4355756638711491) > weight (0.25) 2026-02-22T20:14:41.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3782852152239432) > weight (0.25) 2026-02-22T20:14:41.576 INFO:tasks.rbd_mirror_thrash:waiting for 101.40131540650766 secs before thrashing 2026-02-22T20:14:42.223 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:42.221+0000 7f8da6485640 -1 reset not still connected to 0x55abf552b860 2026-02-22T20:14:42.353 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:42.349+0000 7f8da6485640 -1 reset not still connected to 0x55abea6dcea0 2026-02-22T20:14:42.353 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:42.349+0000 7f8da6485640 -1 reset not still connected to 0x55abf6e176c0 2026-02-22T20:14:42.353 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:42.353+0000 7f8da6485640 -1 reset not still connected to 0x55abf7185e10 2026-02-22T20:14:42.353 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:42.353+0000 7f8da6485640 -1 reset not still connected to 0x55abf96af520 2026-02-22T20:14:43.395 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:43.393+0000 7f8da6485640 -1 reset not still connected to 0x55abea6dcea0 2026-02-22T20:14:43.395 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:43.393+0000 7f8da6485640 -1 reset not still connected to 0x55abf6e176c0 2026-02-22T20:14:43.396 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:43.393+0000 7f8da6485640 -1 reset not still connected to 0x55abf7185e10 2026-02-22T20:14:43.396 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:43.393+0000 7f8da6485640 -1 reset not still connected to 0x55abf96af520 2026-02-22T20:14:43.605 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:43.601+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3332c0d0 2026-02-22T20:14:43.606 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:43.605+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc1110 2026-02-22T20:14:44.949 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:44.945+0000 7f8da6485640 -1 reset not still connected to 0x55abe23bd6c0 2026-02-22T20:14:44.950 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:44.949+0000 7f8da6485640 -1 reset not still connected to 0x55abe5a8ed00 2026-02-22T20:14:44.950 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:44.949+0000 7f8da6485640 -1 reset not still connected to 0x55abea6dcea0 2026-02-22T20:14:44.951 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:44.949+0000 7f8da6485640 -1 reset not still connected to 0x55abf6e176c0 2026-02-22T20:14:44.951 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:44.949+0000 7f8da6485640 -1 reset not still connected to 0x55abf7185e10 2026-02-22T20:14:44.951 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:44.949+0000 7f8da6485640 -1 reset not still connected to 0x55abf96af520 2026-02-22T20:14:45.756 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:45.753+0000 7f8da6485640 -1 reset not still connected to 0x55abc5af1110 2026-02-22T20:14:45.757 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:45.753+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:14:45.757 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:45.753+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:14:45.757 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:45.753+0000 7f8da6485640 -1 reset not still connected to 0x55abcb21a680 2026-02-22T20:14:45.757 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:45.753+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:14:45.757 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:45.753+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:14:45.787 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:45.785+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:14:49.259 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:49.257+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:14:49.259 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:49.257+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:14:49.259 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:49.257+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:14:49.259 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:49.257+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:14:50.128 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.125+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e097ba0 2026-02-22T20:14:50.129 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.125+0000 7f5e092a6640 -1 reset not still connected to 0x55ac2e3904e0 2026-02-22T20:14:50.129 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.125+0000 7f5e092a6640 -1 reset not still connected to 0x55ac30cf7d40 2026-02-22T20:14:50.129 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.125+0000 7f5e092a6640 -1 reset not still connected to 0x55ac313c92b0 2026-02-22T20:14:50.129 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.125+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc20d0 2026-02-22T20:14:50.130 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.129+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fd8e0d0 2026-02-22T20:14:50.130 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.129+0000 7f5e092a6640 -1 reset not still connected to 0x55ac539bfc70 2026-02-22T20:14:50.130 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.129+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c4410 2026-02-22T20:14:50.130 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.129+0000 7f5e092a6640 -1 reset not still connected to 0x55ac552c5d40 2026-02-22T20:14:50.131 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:50.129+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59a36000 2026-02-22T20:14:52.174 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:52.173+0000 7f8da6485640 -1 reset not still connected to 0x55abe23bd6c0 2026-02-22T20:14:52.174 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:52.173+0000 7f8da6485640 -1 reset not still connected to 0x55abe5a8ed00 2026-02-22T20:14:52.174 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:52.173+0000 7f8da6485640 -1 reset not still connected to 0x55abea6dcea0 2026-02-22T20:14:52.175 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:52.173+0000 7f8da6485640 -1 reset not still connected to 0x55abf6e176c0 2026-02-22T20:14:52.175 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:52.173+0000 7f8da6485640 -1 reset not still connected to 0x55abf7185e10 2026-02-22T20:14:52.175 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:52.173+0000 7f8da6485640 -1 reset not still connected to 0x55abf96af520 2026-02-22T20:14:52.898 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:52.897+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:14:53.982 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.981+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:14:53.982 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.981+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:14:53.982 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.981+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:14:53.983 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.981+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:14:53.983 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.981+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:14:53.984 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.981+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:14:53.985 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.981+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:14:53.985 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.981+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:14:53.985 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.985+0000 7f65e934d640 -1 reset not still connected to 0x55f6f832f860 2026-02-22T20:14:53.985 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:53.985+0000 7f65e934d640 -1 reset not still connected to 0x55f6fb4441a0 2026-02-22T20:14:54.854 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:54.853+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3de05450 2026-02-22T20:14:54.855 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:54.853+0000 7f5e092a6640 -1 reset not still connected to 0x55ac415984e0 2026-02-22T20:14:54.855 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:54.853+0000 7f5e092a6640 -1 reset not still connected to 0x55ac415cc5b0 2026-02-22T20:14:54.855 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:54.853+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4df58c30 2026-02-22T20:14:54.856 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:54.853+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4f2e7ba0 2026-02-22T20:14:54.856 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:54.853+0000 7f5e092a6640 -1 reset not still connected to 0x55ac59ee5380 2026-02-22T20:14:54.856 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:54.853+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5a50cd00 2026-02-22T20:14:54.856 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:54.853+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5cdabad0 2026-02-22T20:14:54.856 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:54.853+0000 7f5e092a6640 -1 reset not still connected to 0x55ac646f0340 2026-02-22T20:14:56.536 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4693352157291085) > weight (0.25) 2026-02-22T20:14:56.537 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.648969905784554) > weight (0.25) 2026-02-22T20:14:56.537 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5972502997311874) > weight (0.25) 2026-02-22T20:14:56.537 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7335047720596888) > weight (0.25) 2026-02-22T20:14:56.537 INFO:tasks.rbd_mirror_thrash:waiting for 94.25506263625995 secs before thrashing 2026-02-22T20:14:57.292 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:57.289+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4b3e1110 2026-02-22T20:14:57.293 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:57.289+0000 7f5e092a6640 -1 reset not still connected to 0x55ac515aba00 2026-02-22T20:14:57.293 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:57.289+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:14:57.294 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:14:57.293+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:14:57.390 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:57.389+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0391ee0 2026-02-22T20:14:57.390 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:57.389+0000 7f65e934d640 -1 reset not still connected to 0x55f6e07a0270 2026-02-22T20:14:57.390 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:57.389+0000 7f65e934d640 -1 reset not still connected to 0x55f6e804a750 2026-02-22T20:14:57.390 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:14:57.389+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:14:58.772 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:58.769+0000 7f8da6485640 -1 reset not still connected to 0x55abddd5e5b0 2026-02-22T20:14:58.772 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:58.769+0000 7f8da6485640 -1 reset not still connected to 0x55abe5309e10 2026-02-22T20:14:58.772 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:58.769+0000 7f8da6485640 -1 reset not still connected to 0x55abf552b860 2026-02-22T20:14:59.247 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:59.245+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:14:59.247 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:14:59.245+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:15:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-22T20:15:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep 'image health:' 2026-02-22T20:15:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut '-d ' -f 3 2026-02-22T20:15:10.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: failed to query services: (13) Permission denied 2026-02-22T20:15:10.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=OK 2026-02-22T20:15:10.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test OK = ERROR 2026-02-22T20:15:10.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test OK = OK 2026-02-22T20:15:10.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:10.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ testlog 'TEST: compare many images' 2026-02-22T20:15:10.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ tee -a /tmp/tmp.Xb7Kt1LMAh/rbd-mirror.test.log 2026-02-22T20:15:10.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ date '+%F %T' 2026-02-22T20:15:10.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo 2026-02-22 20:15:10 TEST: compare many images 2026-02-22T20:15:10.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:2026-02-22 20:15:10 TEST: compare many images 2026-02-22T20:15:10.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 50 2026-02-22T20:15:10.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:15:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_1 2026-02-22T20:15:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_1 snap 2026-02-22T20:15:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:10.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:15:11.057 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:11.052+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:15:11.057 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:11.052+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:15:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2026-02-22T20:15:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:12.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2026-02-22T20:15:12.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:12.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:12.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:12.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:15:12.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:15:12.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:12.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:15:12.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:12.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2026-02-22T20:15:12.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:12.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:12.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:12.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:15:12.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:12.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:12.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:12.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2026-02-22T20:15:12.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:15:12.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:15:12.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:15:12.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:15:12.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:15:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:15:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:15:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2026-02-22T20:15:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_1 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:15:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:12.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:12.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:12.956 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:12.952+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3ea981a0 2026-02-22T20:15:12.956 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:12.952+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4b880dd0 2026-02-22T20:15:12.956 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:12.952+0000 7f5e092a6640 -1 reset not still connected to 0x55ac565e0ea0 2026-02-22T20:15:12.957 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:12.952+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:15:12.957 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:12.952+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:15:13.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_1 2026-02-22T20:15:13.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:13.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_1 ']' 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_1' 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:13.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:13.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:13.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:13.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:13.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:13.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:13.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2026-02-22T20:15:13.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:13.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_1 2026-02-22T20:15:13.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:13.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:13.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:13.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_1 '' 2026-02-22T20:15:13.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:13.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:13.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:13.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:13.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_1.status 2026-02-22T20:15:13.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_1.status 2026-02-22T20:15:13.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:13.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2026-02-22T20:15:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:15:13.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:15:13.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:13.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:13.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902], [object_number=37, tag_tid=1, entry_tid=40901], [object_number=36, tag_tid=1, entry_tid=40900]]], state=connected] 2026-02-22T20:15:13.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:15:13.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2026-02-22T20:15:13.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2026-02-22T20:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_1.status 2026-02-22T20:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_1.status 2026-02-22T20:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2026-02-22T20:15:13.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:15:13.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:15:13.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:13.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:13.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902], [object_number=37, tag_tid=1, entry_tid=40901], [object_number=36, tag_tid=1, entry_tid=40900]]], state=connected] 2026-02-22T20:15:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:13.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:13.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=1, entry_tid=40903' '!=' '' 2026-02-22T20:15:13.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:15:13.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=1, entry_tid=40903' = '' 2026-02-22T20:15:13.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:13.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:15:13.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:15:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:13.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:15:13.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:15:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:13.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:15:13.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40903 2026-02-22T20:15:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 40903 2026-02-22T20:15:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:13.381 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:13.376+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3fbc1110 2026-02-22T20:15:13.382 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:13.380+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4b3e1110 2026-02-22T20:15:13.382 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:13.380+0000 7f5e092a6640 -1 reset not still connected to 0x55ac515aba00 2026-02-22T20:15:13.382 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:13.380+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d1e55f0 2026-02-22T20:15:13.384 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:13.380+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d6fec30 2026-02-22T20:15:13.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_1 2026-02-22T20:15:13.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_1 ']' 2026-02-22T20:15:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_1' 2026-02-22T20:15:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2026-02-22T20:15:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:13.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:13.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:13.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:13.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:13.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:13.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2026-02-22T20:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:13.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:13.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:13.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:13.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:13.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:13.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:13.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:13.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:13.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:13.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:13.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:13.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2026-02-22T20:15:13.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:13.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_1 2026-02-22T20:15:13.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:13.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_1 '' 2026-02-22T20:15:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_1.status 2026-02-22T20:15:13.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_1.status 2026-02-22T20:15:13.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:13.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2026-02-22T20:15:13.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:15:13.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:15:13.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:13.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:13.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902], [object_number=37, tag_tid=1, entry_tid=40901], [object_number=36, tag_tid=1, entry_tid=40900]]], state=connected] 2026-02-22T20:15:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_1.status 2026-02-22T20:15:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_1.status 2026-02-22T20:15:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2026-02-22T20:15:13.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:15:13.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:15:13.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:13.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:13.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902], [object_number=37, tag_tid=1, entry_tid=40901], [object_number=36, tag_tid=1, entry_tid=40900]]], state=connected] 2026-02-22T20:15:13.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=1, entry_tid=40903' '!=' 'object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:15:14.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_1 2026-02-22T20:15:14.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:14.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:14.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_1 ']' 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_1' 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:14.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:14.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:14.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:14.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:14.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2026-02-22T20:15:14.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_1 2026-02-22T20:15:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_1 '' 2026-02-22T20:15:14.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:14.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:14.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:14.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:14.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_1.status 2026-02-22T20:15:14.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_1.status 2026-02-22T20:15:14.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:14.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:14.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2026-02-22T20:15:14.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:14.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:15:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:15:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902], [object_number=37, tag_tid=1, entry_tid=40901], [object_number=36, tag_tid=1, entry_tid=40900]]], state=connected] 2026-02-22T20:15:14.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:15:14.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2026-02-22T20:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2026-02-22T20:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_1.status 2026-02-22T20:15:14.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_1.status 2026-02-22T20:15:14.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2026-02-22T20:15:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:14.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:15:14.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:15:14.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:14.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:14.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902], [object_number=37, tag_tid=1, entry_tid=40901], [object_number=36, tag_tid=1, entry_tid=40900]]], state=connected] 2026-02-22T20:15:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:14.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=39, tag_tid=1, entry_tid=40903' '!=' 'object_number=39, tag_tid=1, entry_tid=40903' 2026-02-22T20:15:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:15:14.851 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:14.848+0000 7f65e934d640 -1 reset not still connected to 0x55f6c8ec9a00 2026-02-22T20:15:14.851 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:14.848+0000 7f65e934d640 -1 reset not still connected to 0x55f6cadef450 2026-02-22T20:15:14.851 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:14.848+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:15:14.851 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:14.848+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:15:14.851 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:14.848+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:15:14.852 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:14.848+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:15:14.852 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:14.848+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:15:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_1 2026-02-22T20:15:14.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:14.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:14.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:14.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:14.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_1 ']' 2026-02-22T20:15:14.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_1' 2026-02-22T20:15:14.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2026-02-22T20:15:14.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:14.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:14.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:14.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:14.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:14.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:14.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:14.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:14.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:14.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:14.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:14.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:14.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:14.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:14.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:14.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:14.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2026-02-22T20:15:15.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:15.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_1 2026-02-22T20:15:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_1 '' 2026-02-22T20:15:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_1.status 2026-02-22T20:15:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_1.status 2026-02-22T20:15:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:15.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:15.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2026-02-22T20:15:15.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:15:15.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:15:15.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:15.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:15.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:15.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:15.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:15:15.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2026-02-22T20:15:15.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_1.status 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_1.status 2026-02-22T20:15:15.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:15.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:15.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2026-02-22T20:15:15.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:15:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:15:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=40903], [object_number=38, tag_tid=1, entry_tid=40902]]], state=connected] 2026-02-22T20:15:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.status 2026-02-22T20:15:15.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:15:15.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:16.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2026-02-22T20:15:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_1': 2026-02-22T20:15:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:15:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:15:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:15:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11271a83d43c 2026-02-22T20:15:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11271a83d43c 2026-02-22T20:15:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:15:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:15:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:15:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:15:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:07:02 2026 2026-02-22T20:15:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:07:02 2026 2026-02-22T20:15:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:08:03 2026 2026-02-22T20:15:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:15:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11271a83d43c 2026-02-22T20:15:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:15:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:15:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 597bd27a-1090-4166-aef9-4245c3b87623 2026-02-22T20:15:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:15:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_1 snap 2026-02-22T20:15:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:15:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:15:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.export 2026-02-22T20:15:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_1.export 2026-02-22T20:15:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_1.export 2026-02-22T20:15:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.export 2026-02-22T20:15:16.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:16.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_1.export 2026-02-22T20:15:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:16.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_1.export 2026-02-22T20:15:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_1.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_1.export 2026-02-22T20:15:16.582 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:16.576+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:15:16.582 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:16.580+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:15:16.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:16.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:15:16.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_2 2026-02-22T20:15:16.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_2 snap 2026-02-22T20:15:16.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:16.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:15:18.121 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:18.116+0000 7f65e934d640 -1 reset not still connected to 0x55f6c2e29930 2026-02-22T20:15:18.121 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:18.116+0000 7f65e934d640 -1 reset not still connected to 0x55f6c8ec9a00 2026-02-22T20:15:18.122 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:18.120+0000 7f65e934d640 -1 reset not still connected to 0x55f6cadef450 2026-02-22T20:15:18.123 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:18.120+0000 7f65e934d640 -1 reset not still connected to 0x55f6de7b1110 2026-02-22T20:15:18.123 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:18.120+0000 7f65e934d640 -1 reset not still connected to 0x55f6df52b2b0 2026-02-22T20:15:18.123 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:18.120+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0a14dd0 2026-02-22T20:15:18.123 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:18.120+0000 7f65e934d640 -1 reset not still connected to 0x55f6e7ae09c0 2026-02-22T20:15:18.124 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:18.120+0000 7f65e934d640 -1 reset not still connected to 0x55f6e80c7ee0 2026-02-22T20:15:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2026-02-22T20:15:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2026-02-22T20:15:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:15:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:20.827 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:20.824+0000 7f5e092a6640 -1 reset not still connected to 0x55ac565e0ea0 2026-02-22T20:15:20.828 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:20.824+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:15:20.829 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:20.824+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5c18c4e0 2026-02-22T20:15:20.829 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:20.824+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:15:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2026-02-22T20:15:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:15:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:15:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:15:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2026-02-22T20:15:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:15:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:20.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:15:20.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:15:20.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:15:20.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:20.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:15:20.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:20.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:15:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:20.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:20.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:20.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:20.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:20.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:20.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:15:20.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:15:20.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:20.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:20.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:20.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2026-02-22T20:15:20.905 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:20.900+0000 7f5e092a6640 -1 reset not still connected to 0x55ac3334e8f0 2026-02-22T20:15:20.906 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:20.904+0000 7f5e092a6640 -1 reset not still connected to 0x55ac565e0ea0 2026-02-22T20:15:20.906 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:20.904+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:15:20.906 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:20.904+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:15:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:15:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:15:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:15:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:15:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:15:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:15:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:15:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_2 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:15:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:15:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_2 2026-02-22T20:15:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_2 ']' 2026-02-22T20:15:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_2' 2026-02-22T20:15:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2026-02-22T20:15:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2026-02-22T20:15:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:21.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:21.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:21.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:21.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:21.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:21.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2026-02-22T20:15:21.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:21.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_2 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '' 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:21.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:21.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:21.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:21.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:21.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:21.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:21.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422], [object_number=25, tag_tid=1, entry_tid=28421], [object_number=24, tag_tid=1, entry_tid=28420]]], state=connected] 2026-02-22T20:15:21.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:21.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:21.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:21.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:21.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:21.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:21.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:21.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:21.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:21.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:21.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422], [object_number=25, tag_tid=1, entry_tid=28421], [object_number=24, tag_tid=1, entry_tid=28420]]], state=connected] 2026-02-22T20:15:21.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=27, tag_tid=1, entry_tid=28423' '!=' '' 2026-02-22T20:15:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:15:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=27, tag_tid=1, entry_tid=28423' = '' 2026-02-22T20:15:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:15:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:21.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:15:21.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:21.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:21.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:21.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:15:21.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:15:21.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:21.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:21.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:15:21.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:21.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:21.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:21.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28423 2026-02-22T20:15:21.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 28423 2026-02-22T20:15:21.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:21.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:21.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:21.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_2 2026-02-22T20:15:21.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:21.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:21.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:21.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_2 ']' 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_2' 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:21.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:21.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:21.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:21.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:21.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:21.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:21.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:21.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:21.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:21.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:21.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:21.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:21.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:21.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:21.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2026-02-22T20:15:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_2 2026-02-22T20:15:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '' 2026-02-22T20:15:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:21.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:21.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:21.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:21.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:21.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:21.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422], [object_number=25, tag_tid=1, entry_tid=28421], [object_number=24, tag_tid=1, entry_tid=28420]]], state=connected] 2026-02-22T20:15:21.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:15:21.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2026-02-22T20:15:21.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:21.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:21.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422], [object_number=25, tag_tid=1, entry_tid=28421], [object_number=24, tag_tid=1, entry_tid=28420]]], state=connected] 2026-02-22T20:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:21.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:21.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:21.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=27, tag_tid=1, entry_tid=28423' '!=' 'object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:21.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:21.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:15:22.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_2 2026-02-22T20:15:22.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:22.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:22.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_2 ']' 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_2' 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:22.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:22.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:22.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2026-02-22T20:15:22.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:22.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:22.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:22.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:22.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:22.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:22.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:22.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:22.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:22.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:22.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2026-02-22T20:15:22.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:22.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_2 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '' 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:22.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:22.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:22.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:22.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:22.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:22.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:22.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:22.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422], [object_number=25, tag_tid=1, entry_tid=28421], [object_number=24, tag_tid=1, entry_tid=28420]]], state=connected] 2026-02-22T20:15:22.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:22.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:22.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:22.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:22.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:22.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:22.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422], [object_number=25, tag_tid=1, entry_tid=28421], [object_number=24, tag_tid=1, entry_tid=28420]]], state=connected] 2026-02-22T20:15:22.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:22.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:22.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:22.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=27, tag_tid=1, entry_tid=28423' '!=' 'object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:22.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:22.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:15:22.481 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:22.476+0000 7f5e092a6640 -1 reset not still connected to 0x55ac565e0ea0 2026-02-22T20:15:22.481 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:22.476+0000 7f5e092a6640 -1 reset not still connected to 0x55ac599ec340 2026-02-22T20:15:22.482 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:22.480+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5c18c4e0 2026-02-22T20:15:22.483 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:15:22.480+0000 7f5e092a6640 -1 reset not still connected to 0x55ac5d719450 2026-02-22T20:15:23.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_2 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_2 ']' 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_2' 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:23.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:23.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:23.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:23.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:23.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2026-02-22T20:15:23.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_2 2026-02-22T20:15:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '' 2026-02-22T20:15:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:23.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:23.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:23.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:23.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:23.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422], [object_number=25, tag_tid=1, entry_tid=28421], [object_number=24, tag_tid=1, entry_tid=28420]]], state=connected] 2026-02-22T20:15:23.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:23.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:15:23.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2026-02-22T20:15:23.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:23.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:23.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:23.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2026-02-22T20:15:23.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:23.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:23.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:23.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:23.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:23.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:23.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:23.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:23.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:23.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422], [object_number=25, tag_tid=1, entry_tid=28421], [object_number=24, tag_tid=1, entry_tid=28420]]], state=connected] 2026-02-22T20:15:23.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:23.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:23.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:23.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=27, tag_tid=1, entry_tid=28423' '!=' 'object_number=27, tag_tid=1, entry_tid=28423' 2026-02-22T20:15:23.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:23.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_2 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_2 ']' 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_2' 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:24.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:24.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:24.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2026-02-22T20:15:24.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:24.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_2 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '' 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:24.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:24.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:24.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:24.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:24.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:24.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:24.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:24.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:24.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:24.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:24.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:24.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_2.status 2026-02-22T20:15:24.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_2.status 2026-02-22T20:15:24.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:24.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:24.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2026-02-22T20:15:24.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:24.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:15:24.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:15:24.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:24.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:24.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28423], [object_number=26, tag_tid=1, entry_tid=28422]]], state=connected] 2026-02-22T20:15:24.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.status 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:15:24.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2026-02-22T20:15:25.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_2': 2026-02-22T20:15:25.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:15:25.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:15:25.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:15:25.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aad8ab5a32 2026-02-22T20:15:25.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aad8ab5a32 2026-02-22T20:15:25.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:15:25.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:07:14 2026 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:07:14 2026 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:08:15 2026 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aad8ab5a32 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 6676d8c3-4551-4e7c-b45a-e4e8bf8f6177 2026-02-22T20:15:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:15:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_2 snap 2026-02-22T20:15:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:15:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:15:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.export 2026-02-22T20:15:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_2.export 2026-02-22T20:15:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_2.export 2026-02-22T20:15:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.export 2026-02-22T20:15:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:26.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_2.export 2026-02-22T20:15:26.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:26.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_2.export 2026-02-22T20:15:26.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_2.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_2.export 2026-02-22T20:15:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:15:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_3 2026-02-22T20:15:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_3 snap 2026-02-22T20:15:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:15:28.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:29.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2026-02-22T20:15:29.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:29.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:29.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:29.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:15:29.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:15:29.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:15:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:29.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:29.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:15:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:15:29.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:29.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:29.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:29.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:29.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2026-02-22T20:15:29.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:29.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_32 2026-02-22T20:15:29.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_38 2026-02-22T20:15:29.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_39 2026-02-22T20:15:29.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2026-02-22T20:15:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:15:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2026-02-22T20:15:29.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:29.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_31 2026-02-22T20:15:29.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_35 2026-02-22T20:15:29.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_36 2026-02-22T20:15:29.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:15:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:30.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:30.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:30.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:30.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2026-02-22T20:15:30.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:15:30.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:15:30.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:15:30.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:15:30.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_3 2026-02-22T20:15:30.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:30.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:30.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:30.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:30.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:15:30.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:15:30.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:30.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:30.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_3 2026-02-22T20:15:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:30.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_3 ']' 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_3' 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:30.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:30.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:30.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:30.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:30.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:30.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:30.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:30.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:30.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:30.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:30.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:30.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:30.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:30.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:30.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:30.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_32 2026-02-22T20:15:30.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_38 2026-02-22T20:15:30.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_39 2026-02-22T20:15:30.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:30.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:30.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2026-02-22T20:15:30.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:30.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:30.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:30.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:30.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:30.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:30.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:30.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:30.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_31 2026-02-22T20:15:30.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_35 2026-02-22T20:15:30.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_36 2026-02-22T20:15:30.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:30.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:30.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_3 2026-02-22T20:15:30.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:15:30.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:30.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:30.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:30.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:15:30.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:30.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:30.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:30.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:15:30.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:30.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:30.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:30.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:30.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:30.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:30.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:30.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:30.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:30.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:30.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_3 2026-02-22T20:15:30.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '' 2026-02-22T20:15:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:30.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:30.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:30.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:30.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:30.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36201], [object_number=32, tag_tid=1, entry_tid=36200], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:30.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2026-02-22T20:15:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2026-02-22T20:15:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:30.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:30.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:30.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:30.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:30.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:30.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:30.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:30.707 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:15:30.693+0000 7f557ffc8640 -1 reset not still connected to 0x5592233a1c70 2026-02-22T20:15:30.707 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:15:30.693+0000 7f557ffc8640 -1 reset not still connected to 0x559238297ba0 2026-02-22T20:15:30.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:30.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:30.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36201], [object_number=32, tag_tid=1, entry_tid=36200], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:30.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36201' '!=' '' 2026-02-22T20:15:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:15:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36201' = '' 2026-02-22T20:15:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:30.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:15:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:15:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:30.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:15:30.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:30.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:30.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:30.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36201 2026-02-22T20:15:30.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 36201 2026-02-22T20:15:30.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:30.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:30.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_3 2026-02-22T20:15:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_3 ']' 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_3' 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:30.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:30.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:30.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_32 2026-02-22T20:15:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_38 2026-02-22T20:15:31.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_39 2026-02-22T20:15:31.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:31.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:31.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:31.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:31.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:31.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:31.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:31.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:31.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:31.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:31.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:31.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:31.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:31.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:31.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:31.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_31 2026-02-22T20:15:31.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_35 2026-02-22T20:15:31.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_36 2026-02-22T20:15:31.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:31.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:31.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_3 2026-02-22T20:15:31.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:15:31.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:31.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:31.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:31.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:31.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:31.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:31.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:31.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:31.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_3 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '' 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:31.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:31.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:31.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:31.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:31.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:31.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:31.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:31.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36201], [object_number=32, tag_tid=1, entry_tid=36200], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:31.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:31.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2026-02-22T20:15:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2026-02-22T20:15:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:31.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:31.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:31.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:31.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:31.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:31.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:31.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:31.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:31.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36201], [object_number=32, tag_tid=1, entry_tid=36200], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:31.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:31.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:31.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:31.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36201' '!=' 'object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:31.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:31.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:15:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_3 2026-02-22T20:15:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_3 ']' 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_3' 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:31.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:31.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:31.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:31.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:31.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:31.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:31.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_32 2026-02-22T20:15:31.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_38 2026-02-22T20:15:31.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_39 2026-02-22T20:15:31.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:31.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:31.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:31.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:31.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:31.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:31.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:31.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:31.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:31.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:31.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:31.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_31 2026-02-22T20:15:31.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_35 2026-02-22T20:15:31.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_36 2026-02-22T20:15:31.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_3 2026-02-22T20:15:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:15:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:15:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:31.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:31.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:31.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:31.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:31.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:31.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:31.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:31.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:31.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_3 2026-02-22T20:15:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '' 2026-02-22T20:15:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:31.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:31.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:31.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:31.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:32.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:32.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:32.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:32.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:32.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36201], [object_number=32, tag_tid=1, entry_tid=36200], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2026-02-22T20:15:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2026-02-22T20:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:32.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:32.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:32.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:32.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:32.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:32.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:32.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:32.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36201], [object_number=32, tag_tid=1, entry_tid=36200], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:32.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:32.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:32.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:32.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36201' '!=' 'object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:32.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:32.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_3 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_3 ']' 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_3' 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:33.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:33.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:33.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:33.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_32 2026-02-22T20:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_38 2026-02-22T20:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_39 2026-02-22T20:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:33.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:33.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2026-02-22T20:15:33.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:33.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:33.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:33.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:33.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:33.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:33.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_31 2026-02-22T20:15:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_35 2026-02-22T20:15:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_36 2026-02-22T20:15:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:33.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:33.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_3 2026-02-22T20:15:33.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:15:33.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:33.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:33.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:33.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:33.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:33.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:33.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:33.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:33.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:33.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_3 2026-02-22T20:15:33.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:33.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:33.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:33.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '' 2026-02-22T20:15:33.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:33.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:33.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:33.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:33.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:33.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:33.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:33.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:33.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:33.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:33.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:33.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:33.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:33.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:33.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36201], [object_number=32, tag_tid=1, entry_tid=36200], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:33.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:33.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2026-02-22T20:15:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2026-02-22T20:15:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:33.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:33.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:33.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:33.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:33.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:33.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:33.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:33.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:33.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36201], [object_number=32, tag_tid=1, entry_tid=36200], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:33.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:33.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:33.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:33.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36201' '!=' 'object_number=33, tag_tid=1, entry_tid=36201' 2026-02-22T20:15:33.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:33.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:15:35.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_3 2026-02-22T20:15:35.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:35.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_3 ']' 2026-02-22T20:15:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_3' 2026-02-22T20:15:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2026-02-22T20:15:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2026-02-22T20:15:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:35.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:35.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:35.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:35.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:35.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:35.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:35.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:35.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:35.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:35.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:35.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:35.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:35.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:35.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:35.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:35.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:35.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:35.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:35.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:35.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:35.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_32 2026-02-22T20:15:35.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_38 2026-02-22T20:15:35.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_39 2026-02-22T20:15:35.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2026-02-22T20:15:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:35.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:35.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:35.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:35.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:35.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:35.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:35.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:35.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:35.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:35.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:35.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_31 2026-02-22T20:15:35.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_35 2026-02-22T20:15:35.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_36 2026-02-22T20:15:35.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_3 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:35.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:35.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:35.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:35.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_3 2026-02-22T20:15:35.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:35.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_3 2026-02-22T20:15:35.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:35.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:35.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:35.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '' 2026-02-22T20:15:35.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:35.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:35.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:35.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:35.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:35.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:35.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:35.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:35.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:35.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:35.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:35.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:35.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:35.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:35.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:35.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2026-02-22T20:15:35.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:35.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:35.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:35.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2026-02-22T20:15:35.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:35.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:35.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:15:35.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:35.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_3.status 2026-02-22T20:15:35.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_3.status 2026-02-22T20:15:35.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:35.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:35.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2026-02-22T20:15:35.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36199], [object_number=34, tag_tid=1, entry_tid=36198]]], state=connected] 2026-02-22T20:15:35.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.status 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:15:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:35.754 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.752+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:15:35.754 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.752+0000 7f8da6485640 -1 reset not still connected to 0x55abc5af1110 2026-02-22T20:15:35.755 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.752+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:15:35.755 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.752+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:15:35.755 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.752+0000 7f8da6485640 -1 reset not still connected to 0x55abcb21a680 2026-02-22T20:15:35.755 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.752+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:15:35.756 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.752+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:15:35.797 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.792+0000 7f8da6485640 -1 reset not still connected to 0x55abd9d22000 2026-02-22T20:15:35.797 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.792+0000 7f8da6485640 -1 reset not still connected to 0x55abf96e7040 2026-02-22T20:15:35.797 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:35.792+0000 7f8da6485640 -1 reset not still connected to 0x55abf9739860 2026-02-22T20:15:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2026-02-22T20:15:36.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_3': 2026-02-22T20:15:36.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:15:36.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:15:36.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:15:36.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1171ca2c38d4 2026-02-22T20:15:36.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1171ca2c38d4 2026-02-22T20:15:36.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:15:36.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:15:36.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:15:36.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:15:36.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:07:22 2026 2026-02-22T20:15:36.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:07:22 2026 2026-02-22T20:15:36.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:08:23 2026 2026-02-22T20:15:36.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:15:36.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1171ca2c38d4 2026-02-22T20:15:36.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:15:36.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:15:36.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 9642bdd9-b03e-459f-824a-ab9b7d220f30 2026-02-22T20:15:36.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:15:36.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:36.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_3 snap 2026-02-22T20:15:36.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:36.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:15:36.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:15:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.export 2026-02-22T20:15:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_3.export 2026-02-22T20:15:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_3.export 2026-02-22T20:15:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.export 2026-02-22T20:15:36.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:36.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_3.export 2026-02-22T20:15:36.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:36.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_3.export 2026-02-22T20:15:36.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_3.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_3.export 2026-02-22T20:15:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:15:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_4 2026-02-22T20:15:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_4 snap 2026-02-22T20:15:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:15:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:36.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:37.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:38.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2026-02-22T20:15:38.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:38.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:38.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:15:38.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:15:38.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:15:38.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:15:38.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2026-02-22T20:15:38.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:15:38.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:15:38.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:38.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:38.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:15:38.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:15:38.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:38.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:15:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:38.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:38.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:38.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:38.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2026-02-22T20:15:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_40 2026-02-22T20:15:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_44 2026-02-22T20:15:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_48 2026-02-22T20:15:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:38.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:38.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2026-02-22T20:15:38.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:15:38.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:38.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:38.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:38.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:38.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:15:38.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:38.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:38.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:38.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:38.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2026-02-22T20:15:39.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 2 closest matches: 2026-02-22T20:15:39.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_43 2026-02-22T20:15:39.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status mirror/image_47 2026-02-22T20:15:39.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:39.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:15:39.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:39.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:39.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:39.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:39.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2026-02-22T20:15:39.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:15:39.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:15:39.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:15:39.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:15:39.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:15:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_4 2026-02-22T20:15:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:15:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:15:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:15:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:39.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:39.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_4 2026-02-22T20:15:39.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:39.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_4 ']' 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_4' 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:39.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:39.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:39.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:39.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:39.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:39.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:39.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:39.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:39.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:39.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:39.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:39.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:39.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:39.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:39.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:39.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:39.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:39.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2026-02-22T20:15:39.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 3 closest matches: 2026-02-22T20:15:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_40 2026-02-22T20:15:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_44 2026-02-22T20:15:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_48 2026-02-22T20:15:39.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2026-02-22T20:15:39.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:39.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:39.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:39.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:39.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:39.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:39.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:39.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:39.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:39.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:39.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:39.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2026-02-22T20:15:39.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:no valid command found; 2 closest matches: 2026-02-22T20:15:39.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_43 2026-02-22T20:15:39.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush mirror/image_47 2026-02-22T20:15:39.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_4 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:15:39.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:39.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:39.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_4 2026-02-22T20:15:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:39.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_4 2026-02-22T20:15:39.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_4 '' 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_4.status 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_4.status 2026-02-22T20:15:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:39.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.status 2026-02-22T20:15:39.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2026-02-22T20:15:39.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.status 2026-02-22T20:15:39.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:39.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:39.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:39.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=2, entry_tid=3], [object_number=18, tag_tid=2, entry_tid=2]]], state=connected] 2026-02-22T20:15:39.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=2, entry_tid=3], [object_number=18, tag_tid=2, entry_tid=2]]], state=connected] 2026-02-22T20:15:39.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.status 2026-02-22T20:15:39.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:15:39.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2026-02-22T20:15:39.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:39.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:39.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:15:39.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2026-02-22T20:15:39.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:39.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:39.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:15:39.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:39.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_4.status 2026-02-22T20:15:39.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_4.status 2026-02-22T20:15:39.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.status 2026-02-22T20:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2026-02-22T20:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.status 2026-02-22T20:15:39.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:39.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:39.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:39.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=2, entry_tid=3], [object_number=18, tag_tid=2, entry_tid=2]]], state=connected] 2026-02-22T20:15:39.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=2, entry_tid=3], [object_number=18, tag_tid=2, entry_tid=2]]], state=connected] 2026-02-22T20:15:39.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.status 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=3, entry_tid=1' -a 'object_number=17, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:39.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:15:39.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:40.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2026-02-22T20:15:40.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_4': 2026-02-22T20:15:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:15:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:15:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:15:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1127309b3ec8 2026-02-22T20:15:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1127309b3ec8 2026-02-22T20:15:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:15:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:15:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:15:40.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:15:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:08:08 2026 2026-02-22T20:15:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:08:08 2026 2026-02-22T20:15:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:09:09 2026 2026-02-22T20:15:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:15:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1127309b3ec8 2026-02-22T20:15:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:15:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:15:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 78987822-94e8-4c2a-8a2c-4c283ddfbb9d 2026-02-22T20:15:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:15:40.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:40.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_4 snap 2026-02-22T20:15:40.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:40.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:15:40.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:40.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:15:40.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.export 2026-02-22T20:15:40.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_4.export 2026-02-22T20:15:40.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_4.export 2026-02-22T20:15:40.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.export 2026-02-22T20:15:40.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:40.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_4.export 2026-02-22T20:15:40.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:40.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_4.export 2026-02-22T20:15:40.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_4.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_4.export 2026-02-22T20:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_5 2026-02-22T20:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_5 snap 2026-02-22T20:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:44.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2026-02-22T20:15:44.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:44.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:44.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:44.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:15:44.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:15:44.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:15:44.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2026-02-22T20:15:44.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:15:44.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:44.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:44.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:15:44.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:15:44.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:15:44.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:44.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:15:44.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:44.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:15:44.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2026-02-22T20:15:44.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-22T20:15:44.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:44.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:44.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:44.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:15:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:44.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:44.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:44.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:44.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2026-02-22T20:15:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:15:45.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:15:45.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:15:45.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:15:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:15:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:15:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:15:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2026-02-22T20:15:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_5 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_5 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_5 ']' 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_5' 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:45.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2026-02-22T20:15:45.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-22T20:15:45.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:45.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2026-02-22T20:15:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:45.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:45.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:45.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:45.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:45.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:45.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:45.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:45.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:45.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2026-02-22T20:15:45.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_5 2026-02-22T20:15:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_5 '' 2026-02-22T20:15:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_5.status 2026-02-22T20:15:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_5.status 2026-02-22T20:15:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:45.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2026-02-22T20:15:45.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994], [object_number=17, tag_tid=1, entry_tid=18993], [object_number=16, tag_tid=1, entry_tid=18992]]], state=connected] 2026-02-22T20:15:45.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:15:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2026-02-22T20:15:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2026-02-22T20:15:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:45.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:45.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:45.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_5.status 2026-02-22T20:15:45.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_5.status 2026-02-22T20:15:45.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:45.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2026-02-22T20:15:45.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994], [object_number=17, tag_tid=1, entry_tid=18993], [object_number=16, tag_tid=1, entry_tid=18992]]], state=connected] 2026-02-22T20:15:45.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:45.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:45.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=18995' '!=' '' 2026-02-22T20:15:45.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:15:45.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=18995' = '' 2026-02-22T20:15:45.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:45.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:15:45.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:45.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:45.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:15:45.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:45.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:45.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:45.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:15:45.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:15:45.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:45.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:45.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:15:45.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:45.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:45.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:45.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=18995 2026-02-22T20:15:45.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 18995 2026-02-22T20:15:45.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:45.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:45.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_5 2026-02-22T20:15:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_5 ']' 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_5' 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2026-02-22T20:15:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:45.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:45.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:45.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:45.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:45.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2026-02-22T20:15:45.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-22T20:15:45.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:45.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2026-02-22T20:15:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:45.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:45.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:45.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:45.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:45.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2026-02-22T20:15:45.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:45.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_5 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_5 '' 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_5.status 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_5.status 2026-02-22T20:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:45.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2026-02-22T20:15:45.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994], [object_number=17, tag_tid=1, entry_tid=18993], [object_number=16, tag_tid=1, entry_tid=18992]]], state=connected] 2026-02-22T20:15:45.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:15:45.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2026-02-22T20:15:45.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:45.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:45.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:45.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2026-02-22T20:15:45.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:45.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:45.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:45.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:45.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_5.status 2026-02-22T20:15:45.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_5.status 2026-02-22T20:15:45.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:45.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2026-02-22T20:15:45.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:45.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:45.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:45.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:45.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994], [object_number=17, tag_tid=1, entry_tid=18993], [object_number=16, tag_tid=1, entry_tid=18992]]], state=connected] 2026-02-22T20:15:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=18995' '!=' 'object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:15:46.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_5 2026-02-22T20:15:46.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:46.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:46.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:46.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:46.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_5 ']' 2026-02-22T20:15:46.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_5' 2026-02-22T20:15:46.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2026-02-22T20:15:46.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:46.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:46.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:46.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:46.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:46.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:46.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:46.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:46.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:46.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:46.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2026-02-22T20:15:46.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:46.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:46.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:46.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:46.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:46.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:46.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:46.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:46.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:46.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2026-02-22T20:15:46.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-22T20:15:46.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:46.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:46.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:46.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2026-02-22T20:15:46.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:46.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:46.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:46.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:46.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_5 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_5 '' 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:46.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:46.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_5.status 2026-02-22T20:15:46.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_5.status 2026-02-22T20:15:46.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:46.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:46.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2026-02-22T20:15:46.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:46.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:46.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:46.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:46.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:46.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994], [object_number=17, tag_tid=1, entry_tid=18993], [object_number=16, tag_tid=1, entry_tid=18992]]], state=connected] 2026-02-22T20:15:46.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:46.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:46.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_5.status 2026-02-22T20:15:46.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_5.status 2026-02-22T20:15:46.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:46.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:46.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2026-02-22T20:15:46.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:46.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:46.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:46.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:46.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:46.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994], [object_number=17, tag_tid=1, entry_tid=18993], [object_number=16, tag_tid=1, entry_tid=18992]]], state=connected] 2026-02-22T20:15:46.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:46.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:46.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:46.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=18995' '!=' 'object_number=19, tag_tid=1, entry_tid=18995' 2026-02-22T20:15:46.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:46.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_5 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_5 ']' 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_5' 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:47.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:47.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:47.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2026-02-22T20:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-22T20:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:47.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:47.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2026-02-22T20:15:47.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:47.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:47.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:47.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:47.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:47.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:47.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:47.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:47.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:47.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:47.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:47.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:47.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:47.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:47.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:47.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:47.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:47.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2026-02-22T20:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:47.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_5 2026-02-22T20:15:47.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:47.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:47.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:47.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_5 '' 2026-02-22T20:15:47.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:47.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:47.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:47.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:47.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_5.status 2026-02-22T20:15:47.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_5.status 2026-02-22T20:15:47.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:47.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:47.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2026-02-22T20:15:47.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:47.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:47.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:47.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:47.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:47.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:47.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_5.status 2026-02-22T20:15:47.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_5.status 2026-02-22T20:15:47.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:47.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:47.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:47.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2026-02-22T20:15:47.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:15:47.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:15:47.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:47.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:47.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18995], [object_number=18, tag_tid=1, entry_tid=18994]]], state=connected] 2026-02-22T20:15:47.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.status 2026-02-22T20:15:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:15:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:15:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2026-02-22T20:15:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:15:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2026-02-22T20:15:48.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_5': 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e33bc8acef 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e33bc8acef 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:07:42 2026 2026-02-22T20:15:48.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:07:42 2026 2026-02-22T20:15:48.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:07:42 2026 2026-02-22T20:15:48.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:15:48.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e33bc8acef 2026-02-22T20:15:48.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:15:48.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:15:48.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 2594e20c-1a99-4153-9436-8c701861da2f 2026-02-22T20:15:48.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:15:48.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:48.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_5 snap 2026-02-22T20:15:48.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:48.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:15:48.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:48.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:15:48.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.export 2026-02-22T20:15:48.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_5.export 2026-02-22T20:15:48.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_5.export 2026-02-22T20:15:48.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.export 2026-02-22T20:15:48.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:48.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_5.export 2026-02-22T20:15:48.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:48.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_5.export 2026-02-22T20:15:48.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_5.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_5.export 2026-02-22T20:15:48.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:48.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:15:48.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_6 2026-02-22T20:15:48.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_6 snap 2026-02-22T20:15:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:15:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:15:49.919 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:49.915+0000 7f65e934d640 -1 reset not still connected to 0x55f6f45b84e0 2026-02-22T20:15:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2026-02-22T20:15:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:15:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2026-02-22T20:15:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:15:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:15:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2026-02-22T20:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:15:52.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2026-02-22T20:15:52.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:15:52.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:15:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:15:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:15:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:15:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:15:52.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:52.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2026-02-22T20:15:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2026-02-22T20:15:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:52.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:52.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:52.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:52.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:52.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:52.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:15:52.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:15:52.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:52.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:52.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:52.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:52.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2026-02-22T20:15:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2026-02-22T20:15:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:53.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:53.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2026-02-22T20:15:53.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:15:53.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:53.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:53.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:53.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:15:53.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:15:53.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2026-02-22T20:15:53.057 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.055+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:15:53.057 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.055+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:15:53.058 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.055+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:15:53.059 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.059+0000 7f65e934d640 -1 reset not still connected to 0x55f6d5a19930 2026-02-22T20:15:53.060 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.059+0000 7f65e934d640 -1 reset not still connected to 0x55f6e0611c70 2026-02-22T20:15:53.060 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.059+0000 7f65e934d640 -1 reset not still connected to 0x55f6e4340a90 2026-02-22T20:15:53.060 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.059+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:15:53.060 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.059+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:15:53.060 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.059+0000 7f65e934d640 -1 reset not still connected to 0x55f6f66c5380 2026-02-22T20:15:53.060 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.059+0000 7f65e934d640 -1 reset not still connected to 0x55f6f832f860 2026-02-22T20:15:53.060 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:53.059+0000 7f65e934d640 -1 reset not still connected to 0x55f6fb4441a0 2026-02-22T20:15:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2026-02-22T20:15:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_6 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:15:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:15:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_6 2026-02-22T20:15:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:15:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:15:53.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:15:53.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:15:53.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:15:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:15:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_6 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:15:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:15:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:15:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_6 2026-02-22T20:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_6 ']' 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_6' 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:53.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:53.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2026-02-22T20:15:53.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:53.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:53.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:53.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:53.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2026-02-22T20:15:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2026-02-22T20:15:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2026-02-22T20:15:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:53.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2026-02-22T20:15:53.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2026-02-22T20:15:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_6 2026-02-22T20:15:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:15:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:15:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:15:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:15:53.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:15:53.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:53.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:53.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:53.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:53.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_6 2026-02-22T20:15:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2026-02-22T20:15:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_6 2026-02-22T20:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:15:53.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:53.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:53.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:53.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_6 2026-02-22T20:15:53.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:53.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_6 2026-02-22T20:15:53.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:53.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:53.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:15:53.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_6 '' 2026-02-22T20:15:53.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:53.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:53.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:15:53.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:53.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_6.status 2026-02-22T20:15:53.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_6.status 2026-02-22T20:15:53.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:53.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.status 2026-02-22T20:15:53.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2026-02-22T20:15:53.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.status 2026-02-22T20:15:53.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:53.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:53.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:53.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35647], [object_number=34, tag_tid=1, entry_tid=35646]]], state=connected] 2026-02-22T20:15:53.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35647], [object_number=34, tag_tid=1, entry_tid=35646]]], state=connected] 2026-02-22T20:15:53.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.status 2026-02-22T20:15:53.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2026-02-22T20:15:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:15:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2026-02-22T20:15:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:53.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:53.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:15:53.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:53.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_6.status 2026-02-22T20:15:53.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_6.status 2026-02-22T20:15:53.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:53.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.status 2026-02-22T20:15:53.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2026-02-22T20:15:53.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.status 2026-02-22T20:15:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:15:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:15:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35647], [object_number=34, tag_tid=1, entry_tid=35646]]], state=connected] 2026-02-22T20:15:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35647], [object_number=34, tag_tid=1, entry_tid=35646]]], state=connected] 2026-02-22T20:15:53.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.status 2026-02-22T20:15:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:15:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:54.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2026-02-22T20:15:54.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_6': 2026-02-22T20:15:54.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:15:54.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:15:54.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:15:54.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aa6358d6d4 2026-02-22T20:15:54.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aa6358d6d4 2026-02-22T20:15:54.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:15:54.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:07:47 2026 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:07:47 2026 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:08:48 2026 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aa6358d6d4 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 353428e0-6729-4cf7-87d4-4e1ab1aa116e 2026-02-22T20:15:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:15:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_6 snap 2026-02-22T20:15:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:15:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:15:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.export 2026-02-22T20:15:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_6.export 2026-02-22T20:15:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_6.export 2026-02-22T20:15:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.export 2026-02-22T20:15:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:54.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_6.export 2026-02-22T20:15:54.992 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:54.991+0000 7f65d4ef5640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2026-02-22T20:15:54.992 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:54.991+0000 7f65d76fa640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2026-02-22T20:15:54.993 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:54.991+0000 7f65d66f8640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2026-02-22T20:15:54.993 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:15:54.991+0000 7f65d9eff640 -1 bdev(0x55f6bbba9c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2026-02-22T20:15:55.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:15:55.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_6.export 2026-02-22T20:15:55.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_6.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_6.export 2026-02-22T20:15:55.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:55.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:15:55.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_7 2026-02-22T20:15:55.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_7 snap 2026-02-22T20:15:55.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:55.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:55.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:15:55.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:15:55.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:15:55.348 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:55.347+0000 7f8da6485640 -1 reset not still connected to 0x55abc3fb8dd0 2026-02-22T20:15:55.348 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:55.347+0000 7f8da6485640 -1 reset not still connected to 0x55abc5af1110 2026-02-22T20:15:55.349 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:55.347+0000 7f8da6485640 -1 reset not still connected to 0x55abc7c831e0 2026-02-22T20:15:55.349 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:55.347+0000 7f8da6485640 -1 reset not still connected to 0x55abcad6ba00 2026-02-22T20:15:55.350 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:55.351+0000 7f8da6485640 -1 reset not still connected to 0x55abcb21a680 2026-02-22T20:15:55.350 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:55.351+0000 7f8da6485640 -1 reset not still connected to 0x55abe447b860 2026-02-22T20:15:55.350 INFO:tasks.ceph.cluster1.osd.1.trial042.stderr:2026-02-22T20:15:55.351+0000 7f8da6485640 -1 reset not still connected to 0x55abeab74dd0 2026-02-22T20:15:56.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2026-02-22T20:15:56.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:56.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:56.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:15:56.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2026-02-22T20:15:56.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:56.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:56.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:15:56.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:15:56.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:15:56.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:56.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:15:57.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2026-02-22T20:15:57.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:57.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:57.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:15:57.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:15:57.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:15:57.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:15:57.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2026-02-22T20:15:57.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:15:57.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:15:57.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:57.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:57.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:15:57.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:15:57.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:57.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:15:57.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:57.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:57.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:57.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:57.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:57.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:57.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:57.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:57.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:15:57.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:15:57.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2026-02-22T20:15:57.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2026-02-22T20:15:57.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:57.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:57.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2026-02-22T20:15:57.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:15:57.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:15:57.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:57.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:57.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:57.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:57.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:57.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:57.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:57.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:57.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:15:57.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:57.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:57.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:57.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:57.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:15:57.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:15:57.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:15:57.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:15:57.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:57.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:57.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:57.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:57.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2026-02-22T20:15:58.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:15:58.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:15:58.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:15:58.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:15:58.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:15:58.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:15:58.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:15:58.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:58.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_7 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:15:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:15:58.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:58.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:58.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_7 2026-02-22T20:15:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:15:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_7 ']' 2026-02-22T20:15:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_7' 2026-02-22T20:15:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2026-02-22T20:15:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2026-02-22T20:15:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:58.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:58.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:58.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:58.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:58.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:58.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:58.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2026-02-22T20:15:58.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2026-02-22T20:15:58.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:58.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2026-02-22T20:15:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:58.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:58.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:58.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:58.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2026-02-22T20:15:58.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:58.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_7 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '' 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:15:58.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:58.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:15:58.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:15:58.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:15:58.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:58.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:58.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:58.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:15:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2026-02-22T20:15:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2026-02-22T20:15:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:58.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:58.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:58.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:58.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:15:58.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:15:58.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:58.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:15:58.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:15:58.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:15:58.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:58.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:58.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:58.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:58.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:58.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=29962' '!=' '' 2026-02-22T20:15:58.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:15:58.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=29962' = '' 2026-02-22T20:15:58.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:58.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:15:58.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:58.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:58.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:15:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:15:58.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:58.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:15:58.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:15:58.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:58.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:58.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:15:58.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:58.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:15:58.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:15:58.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29962 2026-02-22T20:15:58.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 29962 2026-02-22T20:15:58.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:15:58.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:58.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:15:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_7 2026-02-22T20:15:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:15:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_7 ']' 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_7' 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:58.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:58.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:58.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:58.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:58.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:58.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:58.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:58.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:58.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:58.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:58.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:58.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:58.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2026-02-22T20:15:58.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2026-02-22T20:15:58.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:58.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:58.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2026-02-22T20:15:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_7 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '' 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:58.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:15:58.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:15:58.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:58.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:15:58.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:15:58.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:15:58.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:58.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:58.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:15:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:15:58.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:58.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:15:58.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:58.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:58.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:58.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=29962' '!=' 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:58.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:58.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:15:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_7 2026-02-22T20:15:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:15:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:15:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:15:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_7 ']' 2026-02-22T20:15:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_7' 2026-02-22T20:15:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2026-02-22T20:15:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:15:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:15:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:15:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:15:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:15:59.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:15:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2026-02-22T20:15:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:59.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:59.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:59.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:59.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2026-02-22T20:15:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2026-02-22T20:15:59.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:15:59.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2026-02-22T20:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2026-02-22T20:15:59.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:15:59.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_7 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '' 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:15:59.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:59.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:59.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:15:59.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:15:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:15:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:59.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:15:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2026-02-22T20:15:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2026-02-22T20:15:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:15:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:15:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:15:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:15:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:15:59.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:15:59.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:15:59.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:59.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:15:59.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:15:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:15:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:15:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:15:59.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:15:59.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:59.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:59.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=29962' '!=' 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:15:59.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:15:59.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:16:00.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_7 2026-02-22T20:16:00.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:00.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:00.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:16:00.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:00.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_7 ']' 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_7' 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:00.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:00.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:00.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:00.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:00.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:00.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:00.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:00.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:00.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:00.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:00.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:00.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:00.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:00.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:00.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2026-02-22T20:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2026-02-22T20:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2026-02-22T20:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:00.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:00.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:00.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:00.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:00.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2026-02-22T20:16:00.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:00.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_7 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '' 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:00.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:00.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:16:00.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:00.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:16:00.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:16:00.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:00.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:00.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:00.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:16:00.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2026-02-22T20:16:00.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:00.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:00.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:00.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2026-02-22T20:16:00.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:00.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:00.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:00.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:00.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:16:00.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:16:00.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:00.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:00.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:16:00.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:00.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:16:00.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:16:00.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:00.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:00.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:00.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:00.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:16:00.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:16:00.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=29962' '!=' 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:16:00.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:00.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_7 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_7 ']' 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_7' 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:02.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:02.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:02.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:02.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:02.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2026-02-22T20:16:02.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2026-02-22T20:16:02.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:02.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2026-02-22T20:16:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:02.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:02.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:02.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:02.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2026-02-22T20:16:02.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:02.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_7 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '' 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:16:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:02.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:02.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:16:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:02.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:16:02.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:16:02.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:02.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:02.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:02.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:02.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:16:02.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:16:02.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:02.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:02.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:16:02.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:02.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:16:02.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:16:02.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:02.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:02.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=29962], [object_number=29, tag_tid=1, entry_tid=29961], [object_number=28, tag_tid=1, entry_tid=29960], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:02.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:02.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:16:02.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:16:02.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=29962' '!=' 'object_number=30, tag_tid=1, entry_tid=29962' 2026-02-22T20:16:02.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:02.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:16:04.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_7 2026-02-22T20:16:04.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_7 ']' 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_7' 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:04.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:04.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:04.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:04.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:04.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:04.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:04.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2026-02-22T20:16:04.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2026-02-22T20:16:04.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:04.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:04.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:04.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:04.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:04.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:04.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:04.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:04.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2026-02-22T20:16:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_7 2026-02-22T20:16:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '' 2026-02-22T20:16:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:04.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:16:04.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:16:04.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:04.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:04.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:16:04.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:16:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:16:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:04.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_7.status 2026-02-22T20:16:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_7.status 2026-02-22T20:16:04.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:04.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:04.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2026-02-22T20:16:04.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:04.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:16:04.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:16:04.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:04.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:04.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=29962], [object_number=31, tag_tid=1, entry_tid=29959]]], state=connected] 2026-02-22T20:16:04.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.status 2026-02-22T20:16:04.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:16:04.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:16:04.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:04.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2026-02-22T20:16:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:16:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:16:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2026-02-22T20:16:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_7': 2026-02-22T20:16:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1171a18b10e0 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1171a18b10e0 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:07:59 2026 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:07:59 2026 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:09:00 2026 2026-02-22T20:16:05.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:16:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1171a18b10e0 2026-02-22T20:16:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:16:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:16:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 9756611c-31a6-4d57-b51e-ac629cb0b409 2026-02-22T20:16:05.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:16:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_7 snap 2026-02-22T20:16:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:16:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:05.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:16:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.export 2026-02-22T20:16:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_7.export 2026-02-22T20:16:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_7.export 2026-02-22T20:16:05.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.export 2026-02-22T20:16:06.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:06.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_7.export 2026-02-22T20:16:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:06.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_7.export 2026-02-22T20:16:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_7.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_7.export 2026-02-22T20:16:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:16:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_8 2026-02-22T20:16:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_8 snap 2026-02-22T20:16:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:16:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2026-02-22T20:16:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2026-02-22T20:16:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:16:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:10.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:16:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:10.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:10.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:16:10.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:16:10.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:10.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:10.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:16:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:10.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:10.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:10.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:10.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2026-02-22T20:16:10.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-22T20:16:10.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:10.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:10.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:10.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2026-02-22T20:16:10.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:16:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:16:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:16:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:16:10.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:16:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2026-02-22T20:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_8 2026-02-22T20:16:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:16:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:16:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:16:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:16:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:16:10.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_8 2026-02-22T20:16:10.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:10.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:10.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:10.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:10.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_8 ']' 2026-02-22T20:16:10.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_8' 2026-02-22T20:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2026-02-22T20:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:10.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:10.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:10.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:10.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2026-02-22T20:16:10.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:10.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:10.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:10.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:10.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:10.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:10.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:10.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:10.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:10.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:10.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:10.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:10.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-22T20:16:10.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:10.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:10.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:11.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:11.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_8 2026-02-22T20:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '' 2026-02-22T20:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:11.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:11.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:11.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:11.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:11.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:11.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:11.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:11.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:11.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:11.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2026-02-22T20:16:11.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2026-02-22T20:16:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:11.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:11.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:11.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:11.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:11.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:11.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:11.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:11.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:11.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:11.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:11.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=1, entry_tid=45558' '!=' '' 2026-02-22T20:16:11.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:16:11.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=1, entry_tid=45558' = '' 2026-02-22T20:16:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:16:11.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:16:11.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:16:11.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:11.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:16:11.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:16:11.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:16:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:16:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:16:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:16:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:16:11.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:16:11.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=45558 2026-02-22T20:16:11.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 45558 2026-02-22T20:16:11.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:16:11.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:11.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:16:11.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_8 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_8 ']' 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_8' 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:11.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2026-02-22T20:16:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:11.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:11.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:11.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:11.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:11.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:11.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-22T20:16:11.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:11.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:11.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:11.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:11.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:11.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:11.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:11.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:11.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:11.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_8 2026-02-22T20:16:11.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:11.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '' 2026-02-22T20:16:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:11.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:11.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:11.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:11.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:11.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:11.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:11.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2026-02-22T20:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2026-02-22T20:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:11.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:11.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:11.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:11.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:11.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:11.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:11.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:11.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:11.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:11.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:11.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:11.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:11.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:11.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=1, entry_tid=45558' '!=' 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:11.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:11.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:16:11.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_8 2026-02-22T20:16:11.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:11.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:11.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_8 ']' 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_8' 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:11.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:11.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:11.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:11.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:11.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2026-02-22T20:16:11.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:11.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:11.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:11.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:11.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:11.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:11.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:11.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:11.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:11.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:11.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:11.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:11.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:11.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:12.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-22T20:16:12.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:12.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2026-02-22T20:16:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:12.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:12.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:12.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:12.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:12.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:12.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:12.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_8 2026-02-22T20:16:12.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:12.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:12.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:12.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '' 2026-02-22T20:16:12.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:12.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:12.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:12.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:12.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:12.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:12.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:12.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:12.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:12.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:12.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:12.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:12.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:12.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:12.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:12.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:12.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2026-02-22T20:16:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2026-02-22T20:16:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:12.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:12.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:12.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:12.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:12.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:12.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:12.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:12.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:12.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=1, entry_tid=45558' '!=' 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:12.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:12.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:16:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_8 2026-02-22T20:16:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_8 ']' 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_8' 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:13.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:13.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:13.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:13.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2026-02-22T20:16:13.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:13.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:13.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:13.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:13.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:13.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:13.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:13.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:13.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:13.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:13.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:13.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:13.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:13.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:13.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-22T20:16:13.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:13.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2026-02-22T20:16:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:13.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:13.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:13.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:13.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:13.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:13.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:13.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:13.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_8 2026-02-22T20:16:13.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '' 2026-02-22T20:16:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:13.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:13.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2026-02-22T20:16:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2026-02-22T20:16:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:13.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:13.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:13.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:13.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:13.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:13.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:13.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:13.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:13.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:13.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:13.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:13.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:13.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:13.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=1, entry_tid=45558' '!=' 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:13.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:13.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:16:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_8 2026-02-22T20:16:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_8 ']' 2026-02-22T20:16:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_8' 2026-02-22T20:16:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2026-02-22T20:16:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:15.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:15.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:15.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2026-02-22T20:16:15.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:15.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:15.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:15.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:15.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:15.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-22T20:16:15.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:15.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:15.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:15.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2026-02-22T20:16:15.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:15.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:15.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:15.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:15.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:15.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:15.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:15.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:15.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_8 2026-02-22T20:16:15.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:15.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:15.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:15.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '' 2026-02-22T20:16:15.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:15.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:15.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:15.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:15.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:15.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:15.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:15.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:15.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:15.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:15.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:15.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:15.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2026-02-22T20:16:15.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:15.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:15.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:15.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2026-02-22T20:16:15.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:15.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:15.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:15.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:15.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:15.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:15.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:15.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:15.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:15.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:15.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:15.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:15.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:15.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:15.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:15.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:15.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=1, entry_tid=45558' '!=' 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:15.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:15.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:16:17.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_8 2026-02-22T20:16:17.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:17.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:17.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:17.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:17.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_8 ']' 2026-02-22T20:16:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_8' 2026-02-22T20:16:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2026-02-22T20:16:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:17.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:17.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:17.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:17.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:17.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:17.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:17.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:17.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:17.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:17.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:17.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-22T20:16:17.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:17.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:17.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:17.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:17.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:17.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_8 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '' 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:17.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:17.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:17.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:17.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:17.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:17.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:17.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:17.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:17.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:17.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:17.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:17.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:17.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:17.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=42, tag_tid=1, entry_tid=45558], [object_number=41, tag_tid=1, entry_tid=45557], [object_number=40, tag_tid=1, entry_tid=45556], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:17.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:17.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:17.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:17.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=42, tag_tid=1, entry_tid=45558' '!=' 'object_number=42, tag_tid=1, entry_tid=45558' 2026-02-22T20:16:17.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:17.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:16:19.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_8 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_8 ']' 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_8' 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:19.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:19.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:19.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-22T20:16:19.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:19.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:19.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:19.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2026-02-22T20:16:19.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:19.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:19.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:19.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2026-02-22T20:16:19.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:19.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_8 2026-02-22T20:16:19.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:19.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:19.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:19.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '' 2026-02-22T20:16:19.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:19.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:19.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:19.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:19.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:19.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:19.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:19.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:19.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:19.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:19.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:19.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:19.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:19.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2026-02-22T20:16:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2026-02-22T20:16:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:16:19.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:19.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_8.status 2026-02-22T20:16:19.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_8.status 2026-02-22T20:16:19.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:19.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:19.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2026-02-22T20:16:19.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=42, tag_tid=1, entry_tid=45558], [object_number=43, tag_tid=1, entry_tid=45555]]], state=connected] 2026-02-22T20:16:19.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.status 2026-02-22T20:16:19.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:19.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:16:19.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:19.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2026-02-22T20:16:19.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:19.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:19.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:19.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:19.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:19.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:16:19.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_8': 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e365959916 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e365959916 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:08:10 2026 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:08:10 2026 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:09:11 2026 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e365959916 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: e9fa78a3-c099-40ec-8289-99d28fd7a46c 2026-02-22T20:16:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_8 snap 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.export 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_8.export 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_8.export 2026-02-22T20:16:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.export 2026-02-22T20:16:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:21.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_8.export 2026-02-22T20:16:21.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:21.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_8.export 2026-02-22T20:16:21.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_8.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_8.export 2026-02-22T20:16:21.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_9 2026-02-22T20:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_9 snap 2026-02-22T20:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:16:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2026-02-22T20:16:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:22.978 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-22T20:16:22.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-22T20:16:22.979 INFO:tasks.rbd_mirror_thrash:waiting for 1.1913971901707443 secs before reviving daemons 2026-02-22T20:16:22.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:16:22.965+0000 7f32437a9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 86306) UID: 1000 2026-02-22T20:16:23.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2026-02-22T20:16:23.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:23.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:23.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:16:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:23.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:16:23.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:23.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:23.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:23.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:16:23.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:16:23.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:23.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:23.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:23.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:23.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2026-02-22T20:16:23.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:23.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:23.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:23.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2026-02-22T20:16:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:16:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:16:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:16:23.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2026-02-22T20:16:23.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:23.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:23.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2026-02-22T20:16:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:16:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:16:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:23.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:23.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:23.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2026-02-22T20:16:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_9 2026-02-22T20:16:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:16:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:16:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:24.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:24.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:24.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:24.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_9 2026-02-22T20:16:24.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:16:24.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:16:24.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:16:24.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:16:24.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:16:24.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:16:24.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:16:24.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_9 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:24.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:16:24.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:16:24.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:16:24.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:24.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:16:24.171 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-22T20:16:24.171 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:16:24.172 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-22T20:16:24.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-22T20:16:24.172 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-22T20:16:24.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-22T20:16:24.173 INFO:tasks.rbd_mirror_thrash:waiting for 102.3299746558701 secs before thrashing 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_9 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_9 ']' 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_9' 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:24.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:24.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:24.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:24.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:24.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:24.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_9 2026-02-22T20:16:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:24.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:24.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:24.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:24.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:24.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:24.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:24.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:24.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:24.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_9 2026-02-22T20:16:24.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:16:24.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:24.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:24.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:24.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:24.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:24.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:24.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_9 2026-02-22T20:16:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:16:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:16:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:24.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:24.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:24.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:24.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:24.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:24.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_9 2026-02-22T20:16:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '' 2026-02-22T20:16:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:24.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:24.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:24.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:24.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:24.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:24.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:24.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:24.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:24.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:24.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:24.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:24.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:24.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2026-02-22T20:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2026-02-22T20:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:24.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:24.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:24.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:24.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=29089' '!=' '' 2026-02-22T20:16:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:16:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=29089' = '' 2026-02-22T20:16:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:16:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:16:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:16:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:16:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:16:24.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:16:24.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:24.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:16:24.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:16:24.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:16:24.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:24.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:16:24.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:16:24.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=29089 2026-02-22T20:16:24.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 29089 2026-02-22T20:16:24.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:16:24.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:24.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:16:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_9 2026-02-22T20:16:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_9 ']' 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_9' 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:24.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:24.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2026-02-22T20:16:24.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:24.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:24.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:24.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:24.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:24.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:24.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:24.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:24.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:24.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:24.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:24.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:24.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:24.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_9 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:24.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:24.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:24.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:24.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:25.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:25.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:25.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_9 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:25.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:25.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:25.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:25.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:25.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:25.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_9 2026-02-22T20:16:25.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:16:25.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:25.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:25.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:25.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:25.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:25.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:25.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:25.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_9 2026-02-22T20:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '' 2026-02-22T20:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:25.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:25.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:25.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:25.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:25.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:25.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:25.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:25.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:25.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:25.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:25.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:25.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:25.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2026-02-22T20:16:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2026-02-22T20:16:25.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:25.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:25.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:25.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:25.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:25.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:25.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:25.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:25.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:25.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:25.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:25.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:25.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:25.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=29089' '!=' 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_9 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_9 ']' 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_9' 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:25.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:25.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:25.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:25.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:25.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:25.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_9 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:25.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:25.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:25.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:25.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:25.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:25.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:25.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:25.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:25.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_9 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:25.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:25.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:25.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_9 2026-02-22T20:16:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:16:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:25.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:25.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:25.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:25.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:25.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:25.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_9 2026-02-22T20:16:25.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:25.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:25.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:25.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '' 2026-02-22T20:16:25.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:25.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:25.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:25.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:25.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:25.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:25.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:25.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:25.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:25.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:26.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:26.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:26.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:26.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:26.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:26.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:26.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:26.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2026-02-22T20:16:26.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:26.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2026-02-22T20:16:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:26.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:26.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:26.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:26.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:26.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:26.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=29089' '!=' 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:26.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:16:26.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_9 2026-02-22T20:16:26.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:26.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:26.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:26.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_9 ']' 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_9' 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:26.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:26.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:26.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:26.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:26.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:26.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:26.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:26.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:26.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_9 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:26.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:26.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:26.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:26.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:26.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:26.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:26.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:26.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_9 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:26.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:26.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:26.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:26.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:26.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:27.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:27.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:27.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:27.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_9 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:16:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:27.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:27.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:27.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:27.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:27.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:27.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_9 2026-02-22T20:16:27.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:27.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:27.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:27.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '' 2026-02-22T20:16:27.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:27.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:27.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:27.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:27.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:27.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:27.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:27.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:27.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:27.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:27.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:27.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:27.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:27.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:27.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:27.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2026-02-22T20:16:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2026-02-22T20:16:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:27.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:27.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:27.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:27.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:27.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:27.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:27.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:27.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:27.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:27.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:27.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:27.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=29089' '!=' 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:27.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:27.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:16:28.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_9 2026-02-22T20:16:28.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:28.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:28.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_9 ']' 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_9' 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:28.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:28.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:28.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:28.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:28.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:28.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:28.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:28.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:28.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:28.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_9 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:28.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:28.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:28.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:28.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:28.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:28.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:28.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:28.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:28.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:28.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:28.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:28.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_9 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:28.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:28.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:28.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:28.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_9 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:16:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:29.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:29.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:29.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:29.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:29.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:29.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_9 2026-02-22T20:16:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '' 2026-02-22T20:16:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:29.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:29.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:29.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:29.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:29.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:29.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2026-02-22T20:16:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2026-02-22T20:16:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:29.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:29.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:29.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:29.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:29.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:29.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:29.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:29.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:29.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:29.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:29.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=29089], [object_number=24, tag_tid=1, entry_tid=29088], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:29.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=29089' '!=' 'object_number=25, tag_tid=1, entry_tid=29089' 2026-02-22T20:16:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:16:30.793 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.26379811395562436) > weight (0.25) 2026-02-22T20:16:30.793 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-22T20:16:30.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-22T20:16:30.794 INFO:tasks.rbd_mirror_thrash:waiting for 9.219080473106585 secs before reviving daemons 2026-02-22T20:16:30.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:16:30.781+0000 7f78c5ef5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 84216) UID: 1000 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_9 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_9 ']' 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_9' 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:31.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:31.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:31.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:31.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:31.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:31.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:31.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:31.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:31.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:31.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:31.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:31.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:31.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:31.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:31.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_9 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:31.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:31.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:31.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:31.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:31.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_9 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:31.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:31.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:31.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:31.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:31.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:31.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:31.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:31.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:31.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:31.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:31.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:31.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:31.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-22T20:16:31.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:31.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:31.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_9 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:16:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:31.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:31.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:31.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:31.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_9 2026-02-22T20:16:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_9 2026-02-22T20:16:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '' 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:31.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:31.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:31.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:31.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:31.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2026-02-22T20:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2026-02-22T20:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_9.status 2026-02-22T20:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_9.status 2026-02-22T20:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:31.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:31.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2026-02-22T20:16:31.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:31.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:31.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:31.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:31.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:31.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=29087], [object_number=26, tag_tid=1, entry_tid=29086]]], state=connected] 2026-02-22T20:16:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.status 2026-02-22T20:16:31.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:31.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:31.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:31.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2026-02-22T20:16:31.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:31.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:31.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:31.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:31.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:31.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:16:31.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_9': 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aa9b1e90ae 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aa9b1e90ae 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:08:23 2026 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:08:23 2026 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:09:24 2026 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aa9b1e90ae 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: d29acf52-b706-4b94-b87c-3a565adebdb6 2026-02-22T20:16:32.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_9 snap 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.export 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_9.export 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_9.export 2026-02-22T20:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.export 2026-02-22T20:16:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:32.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_9.export 2026-02-22T20:16:32.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:32.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_9.export 2026-02-22T20:16:33.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_9.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_9.export 2026-02-22T20:16:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:16:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_10 2026-02-22T20:16:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_10 snap 2026-02-22T20:16:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:16:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:16:36.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:36.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:36.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:16:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:16:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:16:37.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:37.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:37.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:16:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:37.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:37.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:37.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:37.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2026-02-22T20:16:37.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2026-02-22T20:16:37.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:37.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2026-02-22T20:16:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:16:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:16:37.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:37.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:37.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:37.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:16:37.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:37.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:37.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:37.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:37.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:37.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:37.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:37.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:16:37.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:16:37.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:37.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:37.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:37.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:37.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2026-02-22T20:16:37.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:37.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:16:37.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:16:37.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2026-02-22T20:16:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:16:38.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:16:38.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:16:38.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:16:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:16:38.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:16:38.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:16:38.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_10 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:16:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:16:38.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:16:38.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:38.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:16:38.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_10 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_10 ']' 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_10' 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:38.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:38.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:38.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:38.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:38.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:38.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:38.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:38.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:38.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2026-02-22T20:16:38.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:38.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:38.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:38.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:38.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:38.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:38.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:38.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:38.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2026-02-22T20:16:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2026-02-22T20:16:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:16:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2026-02-22T20:16:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:16:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:16:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:16:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:38.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:38.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:38.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:38.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2026-02-22T20:16:38.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:16:38.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:38.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_10 2026-02-22T20:16:38.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:16:38.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:38.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:38.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:38.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:16:38.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:38.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:38.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:38.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:38.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_10 2026-02-22T20:16:38.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:38.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_10 2026-02-22T20:16:38.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:38.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:38.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:16:38.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_10 '' 2026-02-22T20:16:38.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:38.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:38.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:16:38.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:38.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_10.status 2026-02-22T20:16:38.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_10.status 2026-02-22T20:16:38.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:38.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.status 2026-02-22T20:16:38.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2026-02-22T20:16:38.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.status 2026-02-22T20:16:38.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:38.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:38.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:38.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28079], [object_number=26, tag_tid=1, entry_tid=28078]]], state=connected] 2026-02-22T20:16:38.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28079], [object_number=26, tag_tid=1, entry_tid=28078]]], state=connected] 2026-02-22T20:16:38.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.status 2026-02-22T20:16:38.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:38.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2026-02-22T20:16:38.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:38.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:38.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:16:38.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2026-02-22T20:16:38.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:38.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:38.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:16:38.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:38.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_10.status 2026-02-22T20:16:38.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_10.status 2026-02-22T20:16:38.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:38.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.status 2026-02-22T20:16:38.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2026-02-22T20:16:38.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.status 2026-02-22T20:16:38.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:16:38.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:16:38.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:38.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28079], [object_number=26, tag_tid=1, entry_tid=28078]]], state=connected] 2026-02-22T20:16:38.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28079], [object_number=26, tag_tid=1, entry_tid=28078]]], state=connected] 2026-02-22T20:16:38.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.status 2026-02-22T20:16:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:16:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2026-02-22T20:16:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:16:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:16:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:39.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2026-02-22T20:16:39.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_10': 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11277751aeeb 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11277751aeeb 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:08:33 2026 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:08:33 2026 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:09:34 2026 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11277751aeeb 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 95f11888-904b-480d-8d88-acb6e047bbbb 2026-02-22T20:16:39.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:16:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_10 snap 2026-02-22T20:16:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:16:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:16:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.export 2026-02-22T20:16:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_10.export 2026-02-22T20:16:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_10.export 2026-02-22T20:16:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.export 2026-02-22T20:16:39.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_10.export 2026-02-22T20:16:39.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:39.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_10.export 2026-02-22T20:16:39.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_10.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_10.export 2026-02-22T20:16:40.013 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-22T20:16:40.013 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:16:40.013 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-22T20:16:40.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-22T20:16:40.014 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-22T20:16:40.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-22T20:16:40.016 INFO:tasks.rbd_mirror_thrash:waiting for 111.84102907902135 secs before thrashing 2026-02-22T20:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_11 2026-02-22T20:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_11 snap 2026-02-22T20:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:16:42.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2026-02-22T20:16:42.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:42.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:42.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:16:42.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2026-02-22T20:16:42.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:42.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:42.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:16:42.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:16:42.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:42.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:42.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:43.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2026-02-22T20:16:43.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:43.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:43.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:16:43.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:16:43.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:16:43.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:16:43.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2026-02-22T20:16:43.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:16:43.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:16:43.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:43.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:43.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:16:43.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:16:43.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:43.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:16:43.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:43.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:16:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:43.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2026-02-22T20:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:16:43.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:16:43.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:16:43.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:16:43.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:16:43.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:16:43.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:16:43.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:43.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2026-02-22T20:16:43.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_11 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:16:43.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:16:43.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:16:44.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_11 2026-02-22T20:16:44.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:44.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:44.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:16:44.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:16:44.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_11 ']' 2026-02-22T20:16:44.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_11' 2026-02-22T20:16:44.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2026-02-22T20:16:44.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:16:44.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:44.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:44.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:16:44.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:16:44.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:44.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:16:44.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:44.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:16:44.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:16:44.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:44.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:44.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:16:44.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:16:44.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:44.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:44.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2026-02-22T20:16:44.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:44.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_11 2026-02-22T20:16:44.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:44.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:44.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_11 2026-02-22T20:16:44.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_11 '' 2026-02-22T20:16:44.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:44.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:44.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_11 2026-02-22T20:16:44.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:16:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_11.status 2026-02-22T20:16:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_11.status 2026-02-22T20:16:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.status 2026-02-22T20:16:44.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2026-02-22T20:16:44.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.status 2026-02-22T20:16:44.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:16:44.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:16:44.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:44.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15511], [object_number=14, tag_tid=1, entry_tid=15510]]], state=connected] 2026-02-22T20:16:44.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15511], [object_number=14, tag_tid=1, entry_tid=15510]]], state=connected] 2026-02-22T20:16:44.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.status 2026-02-22T20:16:44.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_11 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_11 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:16:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_11.status 2026-02-22T20:16:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_11.status 2026-02-22T20:16:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:16:44.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.status 2026-02-22T20:16:44.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2026-02-22T20:16:44.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.status 2026-02-22T20:16:44.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:16:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:16:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:16:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15511], [object_number=14, tag_tid=1, entry_tid=15510]]], state=connected] 2026-02-22T20:16:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15511], [object_number=14, tag_tid=1, entry_tid=15510]]], state=connected] 2026-02-22T20:16:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.status 2026-02-22T20:16:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:16:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:16:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2026-02-22T20:16:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:16:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:16:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:45.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:16:45.093+0000 7f2e517a9640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x55b6cb794a80 handle_process_entry_safe: failed to commit journal event: (85) Interrupted system call should be restarted 2026-02-22T20:16:45.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:16:45.093+0000 7f2e517a9640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x55b6cb794a80 handle_replay_complete: replay encountered an error: (85) Interrupted system call should be restarted 2026-02-22T20:16:45.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2026-02-22T20:16:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_11': 2026-02-22T20:16:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1171d84405d 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1171d84405d 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:08:41 2026 2026-02-22T20:16:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:08:41 2026 2026-02-22T20:16:45.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:08:41 2026 2026-02-22T20:16:45.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:16:45.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1171d84405d 2026-02-22T20:16:45.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:16:45.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:16:45.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 5a83d58c-2e89-4f2b-9080-0e112b65ff08 2026-02-22T20:16:45.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:16:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_11 snap 2026-02-22T20:16:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:16:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:16:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.export 2026-02-22T20:16:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_11.export 2026-02-22T20:16:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_11.export 2026-02-22T20:16:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.export 2026-02-22T20:16:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:45.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_11.export 2026-02-22T20:16:45.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:16:45.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_11.export 2026-02-22T20:16:45.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_11.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_11.export 2026-02-22T20:16:45.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:16:45.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:16:45.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_12 2026-02-22T20:16:45.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_12 snap 2026-02-22T20:16:45.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:16:45.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:45.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:16:45.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:16:45.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:16:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:16:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:16:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2026-02-22T20:16:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:16:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:16:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:16:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:16:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:16:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:16:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2026-02-22T20:16:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:16:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:16:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:16:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:16:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:16:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:16:50.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:50.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:16:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:16:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:50.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:50.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:50.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:50.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2026-02-22T20:16:50.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:50.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:50.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2026-02-22T20:16:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:16:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:16:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:16:50.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:50.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:50.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:50.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:50.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2026-02-22T20:16:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:51.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:51.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2026-02-22T20:16:51.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:16:51.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:51.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:16:51.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:16:51.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:51.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:51.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:51.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:51.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2026-02-22T20:16:51.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:51.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:51.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:51.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:51.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2026-02-22T20:16:51.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:16:51.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2026-02-22T20:16:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 1 2026-02-22T20:16:52.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:52.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:16:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2026-02-22T20:16:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:52.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:52.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:52.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:52.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:52.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:16:52.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:16:52.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:52.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2026-02-22T20:16:52.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:52.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:52.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2026-02-22T20:16:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:16:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:16:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:52.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:52.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:52.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:52.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2026-02-22T20:16:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2026-02-22T20:16:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:16:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:52.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:52.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:52.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:52.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:52.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:52.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:16:52.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:16:52.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:52.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:52.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:52.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2026-02-22T20:16:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:52.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:52.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2026-02-22T20:16:52.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:16:52.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:52.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:52.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:52.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:52.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:16:52.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:16:52.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:52.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:52.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2026-02-22T20:16:52.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:52.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:52.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 2 2026-02-22T20:16:54.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:54.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:16:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:16:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:54.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:54.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:54.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:54.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2026-02-22T20:16:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2026-02-22T20:16:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:16:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:54.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:16:54.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2026-02-22T20:16:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:16:54.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2026-02-22T20:16:54.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:54.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:54.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2026-02-22T20:16:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:16:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:16:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:54.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:54.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:54.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:54.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2026-02-22T20:16:54.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:54.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:54.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:54.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:54.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 4 2026-02-22T20:16:58.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:16:58.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:16:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2026-02-22T20:16:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:16:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:16:58.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:16:58.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:16:58.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:16:58.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:16:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:16:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2026-02-22T20:16:58.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:58.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:58.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:58.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:58.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2026-02-22T20:16:58.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:16:58.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:58.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:58.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:16:58.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:16:58.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:58.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:58.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:16:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:16:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:16:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:16:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:16:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:16:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:16:58.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:16:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2026-02-22T20:16:58.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:58.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:58.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:58.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:16:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:16:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:16:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:16:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:16:58.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:58.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:58.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:16:58.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2026-02-22T20:16:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:16:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2026-02-22T20:16:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:16:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:16:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:16:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:16:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:16:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:16:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:59.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:16:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2026-02-22T20:16:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:16:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:16:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:16:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:16:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 8 2026-02-22T20:17:07.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:07.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:17:07.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:07.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2026-02-22T20:17:07.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:07.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:07.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:07.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:07.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:17:07.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:17:07.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:07.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:07.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:07.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:07.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2026-02-22T20:17:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:17:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:07.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:17:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:07.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:07.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:07.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:07.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2026-02-22T20:17:07.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:07.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:17:07.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:07.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:07.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2026-02-22T20:17:07.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:17:07.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:07.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:07.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:17:07.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:07.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:07.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:17:07.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:17:07.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2026-02-22T20:17:07.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:07.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:17:07.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2026-02-22T20:17:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:17:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:17:07.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:07.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:07.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:07.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:07.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2026-02-22T20:17:07.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:17:07.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:17:07.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:17:07.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:17:07.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:17:07.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:17:07.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_12 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:07.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_12 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_12 ']' 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_12' 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:07.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2026-02-22T20:17:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:07.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:07.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:07.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:07.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:07.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:07.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:07.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:07.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:07.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:07.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:07.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:07.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:07.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:07.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:07.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2026-02-22T20:17:07.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:07.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:07.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:07.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:07.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:07.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:07.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:07.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:07.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:07.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:07.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:07.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:07.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:07.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:07.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:07.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:07.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:07.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:07.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2026-02-22T20:17:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:07.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:07.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:07.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:07.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:07.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:07.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:07.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:07.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:07.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_12 2026-02-22T20:17:07.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:07.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:07.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:07.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '' 2026-02-22T20:17:07.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:07.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:07.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:07.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:07.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:07.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:07.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:07.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:07.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:07.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:07.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:07.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:07.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:07.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:07.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=1, entry_tid=44821], [object_number=40, tag_tid=1, entry_tid=44820], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:07.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:07.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:07.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:07.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:07.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:07.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:07.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:07.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=1, entry_tid=44821], [object_number=40, tag_tid=1, entry_tid=44820], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:07.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=1, entry_tid=44821' '!=' '' 2026-02-22T20:17:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:17:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=1, entry_tid=44821' = '' 2026-02-22T20:17:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:17:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:17:07.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:07.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:17:07.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:17:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:07.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:17:07.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:17:07.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:17:07.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:07.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:17:07.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:07.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:17:07.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:07.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=44821 2026-02-22T20:17:07.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 44821 2026-02-22T20:17:07.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:17:07.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:07.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_12 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_12 ']' 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_12' 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:08.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:08.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:08.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:08.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:08.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:08.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:08.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:08.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:08.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:08.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:08.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:08.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:08.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2026-02-22T20:17:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:08.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:08.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:08.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:08.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:08.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:08.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:08.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:08.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:08.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2026-02-22T20:17:08.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:08.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2026-02-22T20:17:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:08.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:08.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:08.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_12 2026-02-22T20:17:08.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:08.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:08.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:08.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '' 2026-02-22T20:17:08.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:08.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:08.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:08.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:08.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:08.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:08.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:08.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:08.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:08.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:08.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:08.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:08.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:08.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:08.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=1, entry_tid=44821], [object_number=40, tag_tid=1, entry_tid=44820], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:08.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:08.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:17:08.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2026-02-22T20:17:08.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:08.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:08.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:08.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2026-02-22T20:17:08.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:08.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:08.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:08.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:08.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:08.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:08.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:08.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:08.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:08.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:08.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:08.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=1, entry_tid=44821], [object_number=40, tag_tid=1, entry_tid=44820], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:08.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:08.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=1, entry_tid=44821' '!=' 'object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:08.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_12 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_12 ']' 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_12' 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:08.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:08.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:08.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:08.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:08.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:08.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:08.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:08.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:08.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:08.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:08.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:08.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:08.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:08.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:08.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:08.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:08.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2026-02-22T20:17:08.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:08.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:08.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:08.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:08.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:08.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:08.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:08.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:08.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:08.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:09.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:09.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:09.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:09.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:09.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:09.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:09.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:09.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:09.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:09.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2026-02-22T20:17:09.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:09.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:09.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:09.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:09.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:09.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:09.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:09.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_12 2026-02-22T20:17:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '' 2026-02-22T20:17:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:09.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:09.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:09.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:09.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:09.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:09.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:09.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:09.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:09.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:09.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:09.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:09.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=1, entry_tid=44821], [object_number=40, tag_tid=1, entry_tid=44820], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:09.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:17:09.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2026-02-22T20:17:09.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:09.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:09.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:09.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2026-02-22T20:17:09.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:09.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:09.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:09.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:09.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:09.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:09.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:09.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:09.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:09.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:09.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:09.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:09.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:09.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:09.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=1, entry_tid=44821], [object_number=40, tag_tid=1, entry_tid=44820], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:09.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:09.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:09.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:09.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=1, entry_tid=44821' '!=' 'object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:09.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:09.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:17:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_12 2026-02-22T20:17:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_12 ']' 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_12' 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:10.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:10.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:10.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:10.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:10.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:10.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:10.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:10.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:10.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:10.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2026-02-22T20:17:10.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:10.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:10.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:10.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:10.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:10.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:10.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:10.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:10.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:10.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:10.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:10.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:10.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:10.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:10.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:10.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:10.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:10.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:10.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2026-02-22T20:17:10.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:10.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:10.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:10.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:10.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:10.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:10.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:10.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:10.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:10.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:10.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:10.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:10.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_12 2026-02-22T20:17:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '' 2026-02-22T20:17:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:10.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:10.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:10.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:10.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:10.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=1, entry_tid=44821], [object_number=40, tag_tid=1, entry_tid=44820], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:10.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:10.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:17:10.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2026-02-22T20:17:10.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:10.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:10.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:10.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2026-02-22T20:17:10.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:10.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:10.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:10.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:10.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:10.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:10.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:10.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:10.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:10.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=1, entry_tid=44821], [object_number=40, tag_tid=1, entry_tid=44820], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:10.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=41, tag_tid=1, entry_tid=44821' '!=' 'object_number=41, tag_tid=1, entry_tid=44821' 2026-02-22T20:17:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_12 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_12 ']' 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_12' 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:12.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:12.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:12.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:12.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:12.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:12.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:12.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:12.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2026-02-22T20:17:12.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:12.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:12.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:12.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:12.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:12.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:12.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2026-02-22T20:17:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:12.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:12.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2026-02-22T20:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:12.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:12.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:12.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:12.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:12.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:12.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:12.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-22T20:17:12.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:12.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:12.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:12.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2026-02-22T20:17:12.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:12.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:12.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:12.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:12.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:12.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:12.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:12.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:12.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:12.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:12.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:12.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:12.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:12.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:12.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:12.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:12.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:12.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:12.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:12.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:12.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:12.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:12.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2026-02-22T20:17:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_12 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '' 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:12.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:12.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:12.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:12.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:12.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:12.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:12.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:12.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:12.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:12.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:12.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:12.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:17:12.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2026-02-22T20:17:12.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:12.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:12.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:12.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2026-02-22T20:17:12.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:12.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:12.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:17:12.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:12.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_12.status 2026-02-22T20:17:12.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_12.status 2026-02-22T20:17:12.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:12.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:12.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2026-02-22T20:17:12.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:12.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 10 2026-02-22T20:17:12.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:17:12.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:12.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:12.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=43, tag_tid=1, entry_tid=44819], [object_number=42, tag_tid=1, entry_tid=44818]]], state=connected] 2026-02-22T20:17:12.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.status 2026-02-22T20:17:12.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:17:12.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:17:12.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:12.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2026-02-22T20:17:12.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:12.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:12.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:17:12.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:12.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:17:12.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:17:12.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:17:13.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2026-02-22T20:17:13.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_12': 2026-02-22T20:17:13.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:17:13.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:17:13.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:17:13.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e3f5d8b2e8 2026-02-22T20:17:13.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e3f5d8b2e8 2026-02-22T20:17:13.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:17:13.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:17:13.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:08:47 2026 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:08:47 2026 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:09:48 2026 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e3f5d8b2e8 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 59329fe0-59b4-48e3-8b35-a8461a3b351c 2026-02-22T20:17:13.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:17:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_12 snap 2026-02-22T20:17:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:17:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:17:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.export 2026-02-22T20:17:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_12.export 2026-02-22T20:17:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_12.export 2026-02-22T20:17:13.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.export 2026-02-22T20:17:13.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:17:13.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_12.export 2026-02-22T20:17:13.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:17:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_12.export 2026-02-22T20:17:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_12.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_12.export 2026-02-22T20:17:13.930 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:17:13.925+0000 7f65e934d640 -1 reset not still connected to 0x55f6cf9808f0 2026-02-22T20:17:13.930 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:17:13.925+0000 7f65e934d640 -1 reset not still connected to 0x55f6d09c44e0 2026-02-22T20:17:13.930 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:17:13.929+0000 7f65e934d640 -1 reset not still connected to 0x55f6d4a0f1e0 2026-02-22T20:17:13.930 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:17:13.929+0000 7f65e934d640 -1 reset not still connected to 0x55f6f3ca1d40 2026-02-22T20:17:13.931 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:17:13.929+0000 7f65e934d640 -1 reset not still connected to 0x55f6f54a32b0 2026-02-22T20:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_13 2026-02-22T20:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_13 snap 2026-02-22T20:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:17:17.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2026-02-22T20:17:17.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:17:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2026-02-22T20:17:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:17:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:17:18.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:17:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2026-02-22T20:17:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:17:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:17:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:18.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:17:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:17:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:18.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:17:18.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:18.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:17:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2026-02-22T20:17:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:17:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:17:18.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:18.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:18.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:18.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:18.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2026-02-22T20:17:18.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:18.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:17:18.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:18.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:17:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2026-02-22T20:17:19.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:19.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:17:19.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2026-02-22T20:17:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:17:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:17:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2026-02-22T20:17:19.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:19.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:17:19.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:19.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:19.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:19.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:19.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2026-02-22T20:17:19.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:17:19.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:17:19.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:17:19.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:17:19.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_13 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:19.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:17:19.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_13 2026-02-22T20:17:19.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:19.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:19.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:19.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:19.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_13 ']' 2026-02-22T20:17:19.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_13' 2026-02-22T20:17:19.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2026-02-22T20:17:19.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:19.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:19.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:19.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:19.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:19.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:19.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:19.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:19.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:19.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:19.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:19.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:19.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:19.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:19.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:19.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2026-02-22T20:17:19.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:19.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:19.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:19.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:19.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:19.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:19.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:19.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_13 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:19.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:19.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:19.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:19.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:19.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_13 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:19.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:19.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:19.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:19.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:19.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:19.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:19.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:19.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:19.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:19.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_13 2026-02-22T20:17:19.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:19.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:19.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:19.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '' 2026-02-22T20:17:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:19.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:19.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:19.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=28314], [object_number=25, tag_tid=1, entry_tid=28313], [object_number=24, tag_tid=1, entry_tid=28312], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:19.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:17:19.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2026-02-22T20:17:19.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=28314], [object_number=25, tag_tid=1, entry_tid=28313], [object_number=24, tag_tid=1, entry_tid=28312], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:19.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:19.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:19.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=28314' '!=' '' 2026-02-22T20:17:19.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:17:19.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=28314' = '' 2026-02-22T20:17:19.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:19.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:17:19.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:17:19.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:17:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:17:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:17:19.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:17:19.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:17:19.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:19.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:17:19.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:19.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:17:19.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:19.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28314 2026-02-22T20:17:19.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 28314 2026-02-22T20:17:19.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:17:19.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:19.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:17:19.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_13 2026-02-22T20:17:19.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:19.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:19.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:19.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:19.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_13 ']' 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_13' 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:19.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:19.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:19.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:20.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:20.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:20.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:20.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:20.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:20.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:20.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:20.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:20.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:20.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:20.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:20.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:20.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_13 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:20.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:20.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:20.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:20.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:20.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_13 2026-02-22T20:17:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:20.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:20.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:20.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:20.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:20.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:20.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:20.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:20.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:20.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_13 2026-02-22T20:17:20.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:20.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:20.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:20.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '' 2026-02-22T20:17:20.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:20.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:20.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:20.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:20.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:20.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:20.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:20.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:20.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:20.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:20.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:20.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:20.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:20.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:20.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=28314], [object_number=25, tag_tid=1, entry_tid=28313], [object_number=24, tag_tid=1, entry_tid=28312], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:20.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:20.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:17:20.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2026-02-22T20:17:20.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:20.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:20.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:20.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2026-02-22T20:17:20.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:20.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:20.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:20.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:20.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:20.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:20.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:20.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:20.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:20.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:20.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:20.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:20.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:20.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:20.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=28314], [object_number=25, tag_tid=1, entry_tid=28313], [object_number=24, tag_tid=1, entry_tid=28312], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:20.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:20.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=28314' '!=' 'object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:20.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:17:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_13 2026-02-22T20:17:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_13 ']' 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_13' 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:20.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:20.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:20.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:20.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:20.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:20.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:20.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:20.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:20.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:20.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:20.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:20.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:20.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:20.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:20.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:20.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:20.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:20.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:20.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:20.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:20.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:20.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:20.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:20.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:20.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_13 2026-02-22T20:17:20.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:20.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:20.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:20.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:20.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:20.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:20.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:20.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:20.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:20.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:20.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:20.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:20.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:20.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_13 2026-02-22T20:17:20.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:20.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:20.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:20.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:20.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:20.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:21.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:21.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:21.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:21.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:21.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:21.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_13 2026-02-22T20:17:21.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:21.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:21.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:21.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '' 2026-02-22T20:17:21.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:21.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:21.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:21.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:21.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:21.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:21.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:21.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:21.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:21.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:21.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=28314], [object_number=25, tag_tid=1, entry_tid=28313], [object_number=24, tag_tid=1, entry_tid=28312], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:21.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:21.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:17:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2026-02-22T20:17:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2026-02-22T20:17:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:21.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:21.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:21.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:21.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:21.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:21.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:21.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:21.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=28314], [object_number=25, tag_tid=1, entry_tid=28313], [object_number=24, tag_tid=1, entry_tid=28312], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=28314' '!=' 'object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:17:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_13 2026-02-22T20:17:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_13 ']' 2026-02-22T20:17:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_13' 2026-02-22T20:17:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2026-02-22T20:17:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2026-02-22T20:17:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:21.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:21.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:21.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:21.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:22.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:22.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:22.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:22.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:22.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:22.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:22.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:22.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:22.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:22.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:22.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_13 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:22.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:22.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:22.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:22.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_13 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:22.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:22.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:22.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:22.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:22.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:22.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_13 2026-02-22T20:17:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '' 2026-02-22T20:17:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:22.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:22.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:22.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:22.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:22.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:22.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:22.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:22.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:22.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:22.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=28314], [object_number=25, tag_tid=1, entry_tid=28313], [object_number=24, tag_tid=1, entry_tid=28312], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:22.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:22.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:17:22.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2026-02-22T20:17:22.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:22.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:22.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:22.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2026-02-22T20:17:22.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:22.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:22.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:22.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:22.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:22.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:22.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:22.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:22.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:22.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:22.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:22.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:22.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:22.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:22.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=28314], [object_number=25, tag_tid=1, entry_tid=28313], [object_number=24, tag_tid=1, entry_tid=28312], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:22.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=28314' '!=' 'object_number=26, tag_tid=1, entry_tid=28314' 2026-02-22T20:17:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_13 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_13 ']' 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_13' 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:23.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:23.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:23.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:23.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:23.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:24.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:24.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:24.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:24.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:24.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:24.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:24.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:24.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:24.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_13 2026-02-22T20:17:24.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:17:24.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:24.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:24.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:17:24.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:17:24.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:17:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:17:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:24.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2026-02-22T20:17:24.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:24.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:24.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_13 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:17:24.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:24.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:24.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:17:24.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_13 2026-02-22T20:17:24.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:24.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_13 2026-02-22T20:17:24.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:24.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:24.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:24.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '' 2026-02-22T20:17:24.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:24.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:24.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:24.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:24.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:24.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:24.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:24.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:24.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:24.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_13.status 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_13.status 2026-02-22T20:17:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2026-02-22T20:17:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:24.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:17:24.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:17:24.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:24.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:24.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=28314], [object_number=27, tag_tid=1, entry_tid=28311]]], state=connected] 2026-02-22T20:17:24.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.status 2026-02-22T20:17:24.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:17:24.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2026-02-22T20:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:17:25.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_13': 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aa5fc7e68f 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aa5fc7e68f 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:09:01 2026 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:09:01 2026 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:10:02 2026 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aa5fc7e68f 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: c8c6454a-9bfa-47f6-ac23-c2b4828aa4cf 2026-02-22T20:17:25.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:17:25.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_13 snap 2026-02-22T20:17:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:17:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:17:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.export 2026-02-22T20:17:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_13.export 2026-02-22T20:17:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_13.export 2026-02-22T20:17:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.export 2026-02-22T20:17:25.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:17:25.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_13.export 2026-02-22T20:17:25.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:17:25.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_13.export 2026-02-22T20:17:25.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_13.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_13.export 2026-02-22T20:17:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:17:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_14 2026-02-22T20:17:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_14 snap 2026-02-22T20:17:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:17:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2026-02-22T20:17:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2026-02-22T20:17:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:17:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:17:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:17:28.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:17:28.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:17:28.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:28.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:17:28.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:28.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:17:28.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:17:28.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2026-02-22T20:17:28.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2026-02-22T20:17:28.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:17:28.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:28.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:28.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:28.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:28.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:28.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:17:28.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:17:28.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:28.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:28.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:28.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:28.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2026-02-22T20:17:29.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:17:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:17:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:17:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:17:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:17:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:17:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:17:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_14 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_14 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_14 ']' 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_14' 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:29.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:29.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:29.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:29.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:29.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:29.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:29.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:29.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:29.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:29.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:29.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:29.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:29.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:29.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:29.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:29.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:29.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:29.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2026-02-22T20:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:29.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:29.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:29.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:29.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:29.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:29.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_14 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '' 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:29.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15698' '!=' '' 2026-02-22T20:17:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:17:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15698' = '' 2026-02-22T20:17:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:17:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:29.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:17:29.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:29.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:17:29.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:29.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:17:29.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:29.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:17:29.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:29.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:17:29.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:29.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:17:29.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:29.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=15698 2026-02-22T20:17:29.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 15698 2026-02-22T20:17:29.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:17:29.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:29.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_14 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_14 ']' 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_14' 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:29.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:29.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:29.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:29.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:29.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:29.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2026-02-22T20:17:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2026-02-22T20:17:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:29.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:29.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:29.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:29.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:29.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:29.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:29.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_14 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '' 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:29.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:29.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:29.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:29.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:29.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:29.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:29.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:29.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2026-02-22T20:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2026-02-22T20:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:29.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:29.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:29.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:29.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:29.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15698' '!=' 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:29.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:29.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:17:30.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_14 2026-02-22T20:17:30.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:30.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:30.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:30.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_14 ']' 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_14' 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2026-02-22T20:17:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:30.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:30.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:30.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:30.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:30.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:30.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2026-02-22T20:17:30.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:30.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:30.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:30.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2026-02-22T20:17:30.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:30.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:30.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:30.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:30.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:30.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:30.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:30.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:30.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:30.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:30.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:30.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_14 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '' 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:30.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:30.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:30.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:30.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:30.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:30.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:30.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:30.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:30.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:30.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:30.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:30.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:30.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:30.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:30.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:30.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:30.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15698' '!=' 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:30.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:17:31.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_14 2026-02-22T20:17:31.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:31.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_14 ']' 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_14' 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:31.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:31.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:31.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:31.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:31.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:31.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:31.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:31.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:31.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:31.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:31.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:31.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2026-02-22T20:17:31.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:31.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:31.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:31.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2026-02-22T20:17:31.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:31.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:31.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:31.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:31.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:31.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:31.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:31.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:31.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:31.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:31.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_14 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '' 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:31.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:31.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:31.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:31.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:31.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:31.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:31.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:31.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:31.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:31.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:31.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:31.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:31.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:31.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:31.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15698' '!=' 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:31.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:31.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:17:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_14 2026-02-22T20:17:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_14 ']' 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_14' 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:33.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:33.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:33.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:33.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2026-02-22T20:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:33.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:33.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:33.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:33.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:33.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:33.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:33.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:33.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:33.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:33.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2026-02-22T20:17:33.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:33.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:33.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:33.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2026-02-22T20:17:33.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:33.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:33.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:33.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:33.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:33.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:33.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:33.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:33.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:33.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_14 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '' 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:33.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:33.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:33.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:33.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:33.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:33.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:33.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:33.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:33.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:33.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:33.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:33.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:33.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:33.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:33.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:33.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:33.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:33.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:33.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:33.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:33.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:33.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15698' '!=' 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_14 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_14 ']' 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_14' 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:35.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:35.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:35.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:35.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:35.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:35.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2026-02-22T20:17:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:35.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:35.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:35.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:35.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:35.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_14 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '' 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:35.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:35.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:35.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:35.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:35.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:35.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:35.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:35.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:35.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:35.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:35.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:35.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:35.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2026-02-22T20:17:35.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:35.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:35.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:35.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2026-02-22T20:17:35.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:35.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:35.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:35.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:35.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:35.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:35.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:35.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:35.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:35.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:35.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:35.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:35.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:35.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15698' '!=' 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:35.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:35.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:17:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_14 2026-02-22T20:17:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_14 ']' 2026-02-22T20:17:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_14' 2026-02-22T20:17:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2026-02-22T20:17:37.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:37.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:37.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:37.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:37.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:37.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:37.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2026-02-22T20:17:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:37.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:37.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:37.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:37.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:37.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:37.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2026-02-22T20:17:37.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:37.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:37.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:37.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:37.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:37.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:37.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:37.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:37.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:37.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:37.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:37.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_14 2026-02-22T20:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '' 2026-02-22T20:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:37.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:37.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:37.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:37.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:37.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:37.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:38.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:38.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:38.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:38.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:38.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:38.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2026-02-22T20:17:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2026-02-22T20:17:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:38.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:38.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:38.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:38.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:38.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15698], [object_number=13, tag_tid=1, entry_tid=15697], [object_number=12, tag_tid=1, entry_tid=15696], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:38.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:38.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:38.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:38.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15698' '!=' 'object_number=14, tag_tid=1, entry_tid=15698' 2026-02-22T20:17:38.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:38.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 4 2026-02-22T20:17:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_14 2026-02-22T20:17:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_14 ']' 2026-02-22T20:17:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_14' 2026-02-22T20:17:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2026-02-22T20:17:42.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2026-02-22T20:17:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:42.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:42.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:42.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:42.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:42.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2026-02-22T20:17:42.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:42.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2026-02-22T20:17:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:42.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:42.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:42.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:42.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2026-02-22T20:17:42.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:42.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_14 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '' 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:42.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:42.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:42.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:42.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:42.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:42.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:42.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:42.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:42.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:42.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:42.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2026-02-22T20:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2026-02-22T20:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_14.status 2026-02-22T20:17:42.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_14.status 2026-02-22T20:17:42.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:42.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:42.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:42.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2026-02-22T20:17:42.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:17:42.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:17:42.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:42.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:42.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15698], [object_number=15, tag_tid=1, entry_tid=15695]]], state=connected] 2026-02-22T20:17:42.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.status 2026-02-22T20:17:42.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:42.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:17:42.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:42.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2026-02-22T20:17:42.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:42.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:42.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:42.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:42.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:17:42.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:17:42.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:17:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_14': 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1127ead4409a 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1127ead4409a 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:09:11 2026 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:09:11 2026 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:09:11 2026 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1127ead4409a 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 5c39aa2e-5360-48f9-8677-f2436c941398 2026-02-22T20:17:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:17:43.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:43.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_14 snap 2026-02-22T20:17:43.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:43.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:17:43.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:43.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:17:43.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.export 2026-02-22T20:17:43.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_14.export 2026-02-22T20:17:43.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_14.export 2026-02-22T20:17:43.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.export 2026-02-22T20:17:43.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:17:43.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_14.export 2026-02-22T20:17:43.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_14.export 2026-02-22T20:17:43.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_14.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_14.export 2026-02-22T20:17:43.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:43.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:17:43.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_15 2026-02-22T20:17:43.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_15 snap 2026-02-22T20:17:43.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:43.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:43.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:17:43.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:43.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:17:47.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:47.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:17:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2026-02-22T20:17:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:17:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:17:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:17:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:17:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2026-02-22T20:17:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:17:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:17:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:17:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:17:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:17:48.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:48.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:17:48.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:48.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2026-02-22T20:17:48.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:17:48.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:48.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:48.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:48.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:17:48.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:17:48.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:48.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:48.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:48.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:48.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2026-02-22T20:17:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:17:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:17:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:17:48.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:17:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:17:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_15 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_15 2026-02-22T20:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_15 ']' 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_15' 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:49.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2026-02-22T20:17:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:49.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:49.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:49.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:49.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:49.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:49.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:49.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:49.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:49.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2026-02-22T20:17:49.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:49.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_15 2026-02-22T20:17:49.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:49.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:49.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_15 2026-02-22T20:17:49.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_15 '' 2026-02-22T20:17:49.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:49.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:49.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_15 2026-02-22T20:17:49.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:49.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_15.status 2026-02-22T20:17:49.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_15.status 2026-02-22T20:17:49.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:49.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.status 2026-02-22T20:17:49.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2026-02-22T20:17:49.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.status 2026-02-22T20:17:49.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:17:49.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:17:49.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:49.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=29635], [object_number=30, tag_tid=1, entry_tid=29634]]], state=connected] 2026-02-22T20:17:49.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=29635], [object_number=30, tag_tid=1, entry_tid=29634]]], state=connected] 2026-02-22T20:17:49.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.status 2026-02-22T20:17:49.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:17:49.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2026-02-22T20:17:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_15 2026-02-22T20:17:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2026-02-22T20:17:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_15 2026-02-22T20:17:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_15.status 2026-02-22T20:17:49.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_15.status 2026-02-22T20:17:49.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:49.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.status 2026-02-22T20:17:49.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2026-02-22T20:17:49.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.status 2026-02-22T20:17:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:17:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:17:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=29635], [object_number=30, tag_tid=1, entry_tid=29634]]], state=connected] 2026-02-22T20:17:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=29635], [object_number=30, tag_tid=1, entry_tid=29634]]], state=connected] 2026-02-22T20:17:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.status 2026-02-22T20:17:49.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:17:49.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:17:50.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_15': 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11715de8eb07 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11715de8eb07 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:17:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:09:16 2026 2026-02-22T20:17:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:09:16 2026 2026-02-22T20:17:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:10:17 2026 2026-02-22T20:17:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:17:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11715de8eb07 2026-02-22T20:17:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:17:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:17:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 1fa47011-746c-4d42-9ae9-3f9a23caca7f 2026-02-22T20:17:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:17:50.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:50.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_15 snap 2026-02-22T20:17:50.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:50.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:17:50.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:50.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:17:50.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.export 2026-02-22T20:17:50.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_15.export 2026-02-22T20:17:50.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_15.export 2026-02-22T20:17:50.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.export 2026-02-22T20:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:17:50.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_15.export 2026-02-22T20:17:50.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:17:50.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_15.export 2026-02-22T20:17:50.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_15.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_15.export 2026-02-22T20:17:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:17:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_16 2026-02-22T20:17:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_16 snap 2026-02-22T20:17:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:17:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:17:52.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2026-02-22T20:17:52.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:52.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:52.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:52.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2026-02-22T20:17:52.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:52.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:52.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:52.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:17:52.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:17:52.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:52.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:17:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2026-02-22T20:17:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:17:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:17:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:17:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2026-02-22T20:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:53.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:53.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:53.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:53.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2026-02-22T20:17:53.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-22T20:17:53.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:17:53.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:17:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:53.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:53.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:53.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:53.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2026-02-22T20:17:53.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:17:53.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:17:53.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:17:53.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:17:53.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:17:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:17:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_16 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:17:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:17:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:17:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:17:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_16 2026-02-22T20:17:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_16 ']' 2026-02-22T20:17:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_16' 2026-02-22T20:17:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2026-02-22T20:17:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:54.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:54.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:54.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:54.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:54.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:54.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:54.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:54.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:54.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-22T20:17:54.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:54.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:54.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:54.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:54.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:54.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:54.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:54.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:54.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:54.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_16 2026-02-22T20:17:54.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:54.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:54.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:54.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '' 2026-02-22T20:17:54.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:54.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:54.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:54.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:54.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:54.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:54.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:54.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:54.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:54.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:54.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:54.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:54.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:54.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:17:54.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2026-02-22T20:17:54.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:54.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:54.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:54.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2026-02-22T20:17:54.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:54.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:54.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:54.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:54.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:54.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:54.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:54.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:54.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:54.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:54.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:54.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:54.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:54.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=32, tag_tid=1, entry_tid=36092' '!=' '' 2026-02-22T20:17:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:17:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=32, tag_tid=1, entry_tid=36092' = '' 2026-02-22T20:17:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:17:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:17:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:17:54.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:54.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:17:54.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:54.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:17:54.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:17:54.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:17:54.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:54.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:17:54.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:54.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:17:54.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:17:54.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36092 2026-02-22T20:17:54.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 36092 2026-02-22T20:17:54.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:17:54.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:54.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:17:54.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_16 2026-02-22T20:17:54.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:54.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:54.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:54.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:54.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_16 ']' 2026-02-22T20:17:54.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_16' 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:54.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:54.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:54.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:54.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:54.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:54.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:54.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:54.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:54.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:54.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:54.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:54.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:54.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:54.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:54.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:54.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-22T20:17:54.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:54.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2026-02-22T20:17:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:54.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:54.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:54.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:54.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:54.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_16 2026-02-22T20:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:54.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:54.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '' 2026-02-22T20:17:54.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:54.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:54.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:54.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:54.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:54.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:54.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:54.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:54.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:54.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:17:54.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2026-02-22T20:17:54.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:54.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:54.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:54.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2026-02-22T20:17:54.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:54.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:54.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:54.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:54.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:54.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:54.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:54.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:54.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:54.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:54.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:54.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:54.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=32, tag_tid=1, entry_tid=36092' '!=' 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:17:55.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_16 2026-02-22T20:17:55.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:55.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:55.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_16 ']' 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_16' 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:55.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:55.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:55.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2026-02-22T20:17:55.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:55.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:55.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:55.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:55.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:55.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:55.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:55.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:55.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:55.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:55.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:55.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:55.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:55.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-22T20:17:55.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:55.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:55.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:55.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:55.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:55.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:55.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:55.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_16 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '' 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:55.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:55.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:55.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:55.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:55.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:55.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:55.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:55.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:55.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:17:55.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2026-02-22T20:17:55.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:55.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:55.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:55.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2026-02-22T20:17:55.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:55.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:55.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:55.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:55.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:55.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:55.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:55.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:55.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:55.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:55.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:55.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=32, tag_tid=1, entry_tid=36092' '!=' 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:55.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:55.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_16 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_16 ']' 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_16' 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:56.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:56.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:56.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:56.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:56.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:56.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:56.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-22T20:17:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:56.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:56.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2026-02-22T20:17:56.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:56.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:56.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:56.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:56.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:56.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:56.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:56.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:56.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:56.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:56.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:56.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:56.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_16 2026-02-22T20:17:56.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:56.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:56.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:56.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '' 2026-02-22T20:17:56.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:56.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:56.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:56.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:56.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:56.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:56.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:56.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:56.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:56.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:56.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:56.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:56.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:56.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:56.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:17:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2026-02-22T20:17:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2026-02-22T20:17:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:56.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:56.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:56.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:56.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:56.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:56.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:56.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:56.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:56.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:56.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:56.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:56.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:56.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:56.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=32, tag_tid=1, entry_tid=36092' '!=' 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:56.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:56.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_16 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_16 ']' 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_16' 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:17:58.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:17:58.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:17:58.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:17:58.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:17:58.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:17:58.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:17:58.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:58.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:58.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:58.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:17:58.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:17:58.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:58.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:58.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:17:58.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:17:58.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:17:58.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:58.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:58.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:17:58.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-22T20:17:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:17:58.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:17:58.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:17:58.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2026-02-22T20:17:58.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:17:58.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:17:58.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:58.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:17:58.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:58.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:17:58.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:17:58.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:17:58.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:17:58.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:17:58.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:17:58.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_16 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '' 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:58.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:58.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:58.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:58.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:58.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:58.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:58.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:58.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:17:58.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2026-02-22T20:17:58.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:58.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2026-02-22T20:17:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:17:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:17:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:17:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:17:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:17:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:17:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:17:58.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:58.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:17:58.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:58.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:17:58.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:17:58.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:17:58.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:17:58.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:17:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:17:58.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:58.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:58.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=32, tag_tid=1, entry_tid=36092' '!=' 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:17:58.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:17:58.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:18:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_16 2026-02-22T20:18:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_16 ']' 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_16' 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:00.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:00.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:00.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:00.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:00.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2026-02-22T20:18:00.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-22T20:18:00.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:00.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:00.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:00.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:00.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:00.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:00.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:00.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:00.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:00.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:00.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:00.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:00.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2026-02-22T20:18:00.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_16 2026-02-22T20:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '' 2026-02-22T20:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:18:00.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2026-02-22T20:18:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:18:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2026-02-22T20:18:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:18:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:18:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:18:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:18:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:00.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:00.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:00.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:00.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:18:00.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=32, tag_tid=1, entry_tid=36092], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090], [object_number=33, tag_tid=1, entry_tid=36089]]], state=connected] 2026-02-22T20:18:00.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:00.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:18:00.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:18:00.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=32, tag_tid=1, entry_tid=36092' '!=' 'object_number=32, tag_tid=1, entry_tid=36092' 2026-02-22T20:18:00.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:00.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:18:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_16 2026-02-22T20:18:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:18:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_16 ']' 2026-02-22T20:18:02.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_16' 2026-02-22T20:18:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2026-02-22T20:18:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:02.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:02.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:02.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:02.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:02.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2026-02-22T20:18:02.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-22T20:18:02.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:02.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:02.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2026-02-22T20:18:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:02.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:02.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:02.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:02.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:02.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:02.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:02.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:02.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:02.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:02.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:02.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:02.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:02.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:02.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:02.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:02.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:02.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:02.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2026-02-22T20:18:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_16 2026-02-22T20:18:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:18:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '' 2026-02-22T20:18:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:18:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:18:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:18:02.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:02.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:02.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:18:02.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:02.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:02.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:02.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:02.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:18:02.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:18:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:02.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:02.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2026-02-22T20:18:02.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:02.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:02.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:18:02.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2026-02-22T20:18:02.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:02.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:02.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:18:02.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:02.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_16.status 2026-02-22T20:18:02.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_16.status 2026-02-22T20:18:02.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:02.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:02.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2026-02-22T20:18:02.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:18:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36091], [object_number=34, tag_tid=1, entry_tid=36090]]], state=connected] 2026-02-22T20:18:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.status 2026-02-22T20:18:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2026-02-22T20:18:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:18:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:18:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2026-02-22T20:18:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_16': 2026-02-22T20:18:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:18:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:18:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:18:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e3fc22d540 2026-02-22T20:18:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e3fc22d540 2026-02-22T20:18:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:18:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:09:25 2026 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:09:25 2026 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:10:26 2026 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e3fc22d540 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: b19cb865-7819-4796-aa5c-b822787f63f3 2026-02-22T20:18:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:18:03.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:03.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_16 snap 2026-02-22T20:18:03.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:03.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:18:03.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:18:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.export 2026-02-22T20:18:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_16.export 2026-02-22T20:18:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_16.export 2026-02-22T20:18:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.export 2026-02-22T20:18:04.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_16.export 2026-02-22T20:18:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_16.export 2026-02-22T20:18:04.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_16.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_16.export 2026-02-22T20:18:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:18:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_17 2026-02-22T20:18:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_17 snap 2026-02-22T20:18:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:18:06.503 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8151636468806225) > weight (0.25) 2026-02-22T20:18:06.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4571402538457471) > weight (0.25) 2026-02-22T20:18:06.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9176959021958576) > weight (0.25) 2026-02-22T20:18:06.504 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2026-02-22T20:18:06.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2026-02-22T20:18:06.504 INFO:tasks.rbd_mirror_thrash:waiting for 7.937803183798708 secs before reviving daemons 2026-02-22T20:18:06.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:18:06.493+0000 7f82df01f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 20131) UID: 1000 2026-02-22T20:18:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2026-02-22T20:18:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2026-02-22T20:18:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:07.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:18:07.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:07.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:07.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2026-02-22T20:18:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:18:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:18:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:18:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2026-02-22T20:18:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:18:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:18:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:18:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:18:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:18:08.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:08.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:18:08.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:08.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:18:08.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:18:08.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:08.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:08.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:08.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:08.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2026-02-22T20:18:08.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-22T20:18:08.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:18:08.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2026-02-22T20:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:08.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:08.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:08.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:08.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2026-02-22T20:18:08.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:18:08.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:18:08.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:18:08.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:18:08.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:18:08.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_17 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_17 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_17 ']' 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_17' 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2026-02-22T20:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:09.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:09.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:09.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:09.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:09.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:09.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:09.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-22T20:18:09.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:09.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:09.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:09.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2026-02-22T20:18:09.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:09.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:09.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:09.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:09.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:09.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:09.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:09.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:09.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_17 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '' 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:09.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:09.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:09.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:09.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:09.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:09.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2026-02-22T20:18:09.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:09.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:09.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:09.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2026-02-22T20:18:09.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:09.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:09.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:09.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:09.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:09.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:09.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:09.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:09.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=22, tag_tid=1, entry_tid=23998' '!=' '' 2026-02-22T20:18:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:18:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=22, tag_tid=1, entry_tid=23998' = '' 2026-02-22T20:18:09.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:09.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:09.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:09.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:09.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:18:09.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:09.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:09.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:09.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:18:09.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:09.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:09.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:18:09.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:09.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:09.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:09.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=23998 2026-02-22T20:18:09.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 23998 2026-02-22T20:18:09.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:09.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:09.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_17 2026-02-22T20:18:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_17 ']' 2026-02-22T20:18:09.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_17' 2026-02-22T20:18:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2026-02-22T20:18:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:09.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2026-02-22T20:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-22T20:18:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2026-02-22T20:18:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:09.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:09.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:09.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_17 2026-02-22T20:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '' 2026-02-22T20:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:09.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:09.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2026-02-22T20:18:09.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:09.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:09.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:09.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2026-02-22T20:18:09.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:09.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:09.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:09.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:09.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:09.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:09.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:09.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:09.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:09.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=22, tag_tid=1, entry_tid=23998' '!=' 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:18:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_17 2026-02-22T20:18:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_17 ']' 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_17' 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:10.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:10.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:10.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2026-02-22T20:18:10.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:10.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:10.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:10.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:10.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-22T20:18:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2026-02-22T20:18:10.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:10.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:10.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:10.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:10.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:10.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:10.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:10.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:10.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:10.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:10.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_17 2026-02-22T20:18:10.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:10.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:10.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:10.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '' 2026-02-22T20:18:10.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:10.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:10.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:10.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:10.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:10.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:10.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:10.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:10.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:10.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:10.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:10.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:10.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:10.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:10.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:10.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:10.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:10.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:10.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:10.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:10.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:10.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:10.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=22, tag_tid=1, entry_tid=23998' '!=' 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:10.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:10.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:18:11.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_17 2026-02-22T20:18:11.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_17 ']' 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_17' 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:11.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:11.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:11.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:11.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:11.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:11.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:11.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:11.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:11.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:11.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:11.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:11.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-22T20:18:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:11.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:11.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:11.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2026-02-22T20:18:11.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:11.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:11.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:11.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:11.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:11.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:11.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:11.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_17 2026-02-22T20:18:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:11.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '' 2026-02-22T20:18:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:11.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:11.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:11.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:11.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:11.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:11.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:11.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:11.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:11.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:11.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:11.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:11.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:11.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:11.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:11.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:11.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:11.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:11.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:11.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:11.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:11.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:11.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:11.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:11.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=22, tag_tid=1, entry_tid=23998' '!=' 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:18:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_17 2026-02-22T20:18:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_17 ']' 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_17' 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:13.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:13.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:13.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:13.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:13.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-22T20:18:13.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:13.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2026-02-22T20:18:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:13.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:13.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:13.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:13.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:13.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:13.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_17 2026-02-22T20:18:13.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:13.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:13.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:13.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '' 2026-02-22T20:18:13.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:13.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:13.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:13.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:13.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:13.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:13.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2026-02-22T20:18:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2026-02-22T20:18:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:13.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:13.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:13.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:13.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:13.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:13.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:13.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:13.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:13.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=22, tag_tid=1, entry_tid=23998], [object_number=21, tag_tid=1, entry_tid=23997], [object_number=20, tag_tid=1, entry_tid=23996], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=22, tag_tid=1, entry_tid=23998' '!=' 'object_number=22, tag_tid=1, entry_tid=23998' 2026-02-22T20:18:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:18:14.443 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2026-02-22T20:18:14.444 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:18:14.444 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2026-02-22T20:18:14.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2026-02-22T20:18:14.444 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2026-02-22T20:18:14.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2026-02-22T20:18:14.447 INFO:tasks.rbd_mirror_thrash:waiting for 119.12275455472226 secs before thrashing 2026-02-22T20:18:15.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_17 2026-02-22T20:18:15.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:15.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:15.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:15.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:15.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_17 ']' 2026-02-22T20:18:15.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_17' 2026-02-22T20:18:15.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2026-02-22T20:18:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:15.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:15.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:15.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:15.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:15.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:15.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:15.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:15.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-22T20:18:15.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:15.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:15.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:15.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2026-02-22T20:18:15.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:15.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:15.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:15.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:15.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:15.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2026-02-22T20:18:15.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_17 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '' 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:15.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:15.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:15.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:15.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:15.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:15.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:15.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:15.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:15.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:15.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:15.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:15.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:15.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:15.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2026-02-22T20:18:15.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:15.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:15.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:15.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2026-02-22T20:18:15.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:15.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:15.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:18:15.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:15.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_17.status 2026-02-22T20:18:15.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_17.status 2026-02-22T20:18:15.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:15.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:15.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2026-02-22T20:18:15.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:15.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:18:15.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:18:15.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:15.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:15.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=22, tag_tid=1, entry_tid=23998], [object_number=23, tag_tid=1, entry_tid=23995]]], state=connected] 2026-02-22T20:18:15.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.status 2026-02-22T20:18:15.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:18:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:16.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2026-02-22T20:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_17': 2026-02-22T20:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aab75cbb0f 2026-02-22T20:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aab75cbb0f 2026-02-22T20:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:09:36 2026 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:09:36 2026 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:10:37 2026 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aab75cbb0f 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 1a8fdda6-5291-4756-831a-78d8a1c84e61 2026-02-22T20:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_17 snap 2026-02-22T20:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.export 2026-02-22T20:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_17.export 2026-02-22T20:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_17.export 2026-02-22T20:18:16.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.export 2026-02-22T20:18:16.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:16.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_17.export 2026-02-22T20:18:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:16.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_17.export 2026-02-22T20:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_17.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_17.export 2026-02-22T20:18:17.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:17.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:18:17.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_18 2026-02-22T20:18:17.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_18 snap 2026-02-22T20:18:17.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:17.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:17.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:17.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:17.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:18:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2026-02-22T20:18:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2026-02-22T20:18:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:18:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:20.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2026-02-22T20:18:20.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:20.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:20.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:20.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:18:20.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:18:20.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:18:20.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2026-02-22T20:18:20.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:18:20.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:18:20.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:20.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:18:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:18:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:18:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2026-02-22T20:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:18:20.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:20.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:20.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:20.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:20.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2026-02-22T20:18:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:20.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:18:20.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:20.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:20.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2026-02-22T20:18:20.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:18:20.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:20.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:18:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:18:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:20.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:20.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:20.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:20.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:18:20.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:18:20.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2026-02-22T20:18:20.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:20.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:18:20.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:20.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:20.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2026-02-22T20:18:20.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:18:20.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:20.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:20.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:20.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:20.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:20.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:20.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:20.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:18:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:20.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:20.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:20.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:20.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2026-02-22T20:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:18:21.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_18 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:18:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:18:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:21.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_18 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_18 ']' 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_18' 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:21.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:21.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2026-02-22T20:18:21.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:21.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:21.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:21.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:21.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:21.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:21.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:21.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:21.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:21.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:21.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:21.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2026-02-22T20:18:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:21.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:21.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:21.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:21.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:21.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:21.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:21.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:21.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:21.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:21.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:21.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_18 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:21.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:21.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:21.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:21.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_18 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '' 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:21.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:21.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:21.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:21.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:21.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:21.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:21.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:21.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:21.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:21.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:21.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:21.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:21.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:21.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:21.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2026-02-22T20:18:21.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:21.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:21.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:21.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2026-02-22T20:18:21.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:21.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:21.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:21.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:21.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:21.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:21.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:21.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:21.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:21.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:21.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:21.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:21.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:21.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:21.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:21.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:21.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36693' '!=' '' 2026-02-22T20:18:21.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:18:21.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36693' = '' 2026-02-22T20:18:21.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:21.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:21.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:21.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:21.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:18:21.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:21.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:21.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:21.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:18:21.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:21.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:21.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:21.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:18:21.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:21.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:21.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=36693 2026-02-22T20:18:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 36693 2026-02-22T20:18:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:21.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:21.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_18 2026-02-22T20:18:21.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_18 ']' 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_18' 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:21.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:21.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:21.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:21.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2026-02-22T20:18:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:21.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:21.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:21.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:21.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:21.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:21.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:21.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:21.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:21.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:21.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:21.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:21.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:21.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:21.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:21.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:21.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:21.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:21.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:21.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:21.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:21.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:21.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_18 2026-02-22T20:18:21.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:21.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_18 2026-02-22T20:18:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '' 2026-02-22T20:18:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:21.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:21.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:22.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:22.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:22.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:22.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:22.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:22.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:22.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:22.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:22.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:22.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:22.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36693' '!=' 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:22.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:22.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:18:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_18 2026-02-22T20:18:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_18 ']' 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_18' 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:22.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:22.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:22.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:22.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:22.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:22.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:22.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:22.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:22.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:22.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:22.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:22.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:22.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:22.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:22.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:22.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:22.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2026-02-22T20:18:22.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:22.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:22.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:22.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:22.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:22.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:22.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:22.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:22.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:22.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:22.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:22.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_18 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:22.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:22.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:22.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:22.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_18 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '' 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:22.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:22.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:22.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:22.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:22.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2026-02-22T20:18:22.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:22.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:22.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36693' '!=' 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:18:23.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_18 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_18 ']' 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_18' 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:23.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:23.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:23.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:23.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:23.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:23.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:23.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:23.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:23.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:23.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:23.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:23.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:23.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:23.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:23.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:23.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_18 2026-02-22T20:18:23.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:23.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:23.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:23.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:23.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:23.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:23.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:23.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:23.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:23.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:23.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:23.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_18 2026-02-22T20:18:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '' 2026-02-22T20:18:23.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:23.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:23.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:23.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:23.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:23.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:23.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:23.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:23.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:23.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:23.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:23.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:23.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:23.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:23.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:23.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2026-02-22T20:18:23.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:23.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:23.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:23.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2026-02-22T20:18:23.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:23.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:23.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:23.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:23.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:23.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:23.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:23.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:23.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:23.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36693' '!=' 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:18:25.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_18 2026-02-22T20:18:25.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:25.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:25.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:25.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:25.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_18 ']' 2026-02-22T20:18:25.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_18' 2026-02-22T20:18:25.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2026-02-22T20:18:25.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:25.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:25.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:25.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:25.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:25.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:25.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:25.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:25.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:25.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:25.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:25.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:25.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:25.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:25.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:25.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:25.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:25.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2026-02-22T20:18:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:25.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:25.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:25.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:25.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:25.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:25.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:25.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_18 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:25.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:25.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:25.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:25.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:25.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:25.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:25.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:25.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_18 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '' 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:25.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:25.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:25.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:25.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:25.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:25.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:25.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:25.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:25.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2026-02-22T20:18:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2026-02-22T20:18:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:25.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:25.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:25.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:25.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:25.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:25.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:25.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:25.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:25.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:25.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:25.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:25.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36693' '!=' 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:18:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_18 2026-02-22T20:18:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_18 ']' 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_18' 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:27.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:27.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:27.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:27.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:27.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:27.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:27.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:27.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:27.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:27.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:27.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:27.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:27.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:27.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:27.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:27.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:27.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:27.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:27.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:27.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:27.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:27.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_18 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:27.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:27.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:27.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:27.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:27.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_18 2026-02-22T20:18:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '' 2026-02-22T20:18:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:27.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:27.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:27.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:27.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:27.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:27.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:27.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:28.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:28.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:28.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:28.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:28.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:28.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:28.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:28.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:28.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:28.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:28.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:28.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=36693], [object_number=32, tag_tid=1, entry_tid=36692], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:28.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:28.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:28.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:28.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=36693' '!=' 'object_number=33, tag_tid=1, entry_tid=36693' 2026-02-22T20:18:28.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:28.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_18 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_18 ']' 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_18' 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:30.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:30.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:30.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:30.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:30.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:30.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:30.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:30.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:30.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:30.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:30.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:30.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:30.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:30.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:30.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:30.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:30.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:30.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:30.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:30.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2026-02-22T20:18:30.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:30.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_18 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:30.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:30.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:30.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:30.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_18 2026-02-22T20:18:30.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_18 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '' 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:30.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:30.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:30.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:30.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:30.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:30.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:30.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:30.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:30.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:30.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:30.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:30.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:30.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2026-02-22T20:18:30.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:30.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:30.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:30.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2026-02-22T20:18:30.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:30.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:30.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:18:30.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:30.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_18.status 2026-02-22T20:18:30.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_18.status 2026-02-22T20:18:30.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:30.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2026-02-22T20:18:30.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:30.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:30.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:30.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:30.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:30.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=36691], [object_number=34, tag_tid=1, entry_tid=36690]]], state=connected] 2026-02-22T20:18:30.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.status 2026-02-22T20:18:30.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:30.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:30.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:30.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2026-02-22T20:18:30.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:30.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:30.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:30.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:30.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:30.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:18:30.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:31.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_18': 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1127b1d19c5 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1127b1d19c5 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:09:45 2026 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:09:45 2026 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:10:46 2026 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1127b1d19c5 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 7df7c270-c03c-4e13-833e-3c0938fc30ba 2026-02-22T20:18:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:18:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_18 snap 2026-02-22T20:18:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:18:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:18:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.export 2026-02-22T20:18:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_18.export 2026-02-22T20:18:31.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_18.export 2026-02-22T20:18:31.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.export 2026-02-22T20:18:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:31.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_18.export 2026-02-22T20:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:31.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_18.export 2026-02-22T20:18:31.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_18.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_18.export 2026-02-22T20:18:31.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:31.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:18:31.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_19 2026-02-22T20:18:31.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_19 snap 2026-02-22T20:18:31.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:31.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:31.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:18:31.858 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5266268376043514) > weight (0.25) 2026-02-22T20:18:31.858 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5164664372562562) > weight (0.25) 2026-02-22T20:18:31.858 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2026-02-22T20:18:31.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2026-02-22T20:18:31.859 INFO:tasks.rbd_mirror_thrash:waiting for 8.563793255058503 secs before reviving daemons 2026-02-22T20:18:31.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:18:31.845+0000 7f2e587b7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 55805) UID: 1000 2026-02-22T20:18:33.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2026-02-22T20:18:33.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:34.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2026-02-22T20:18:34.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:34.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:18:34.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:34.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:18:34.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:34.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2026-02-22T20:18:34.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:18:34.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:34.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:34.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:34.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:34.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2026-02-22T20:18:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2026-02-22T20:18:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:18:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:18:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:18:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:34.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:34.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:34.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:34.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2026-02-22T20:18:34.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2026-02-22T20:18:34.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:18:34.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:34.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:34.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:18:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:18:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:34.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:34.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:34.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:34.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2026-02-22T20:18:35.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_19 2026-02-22T20:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:35.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:35.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:35.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:35.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_19 2026-02-22T20:18:35.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:18:35.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:18:35.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:18:35.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:18:35.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_19 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:35.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_19 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_19 ']' 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_19' 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:35.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:35.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:35.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:35.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:35.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:35.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:35.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:35.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:35.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2026-02-22T20:18:35.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:35.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:35.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:35.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2026-02-22T20:18:35.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:35.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:35.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:35.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:35.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:35.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:35.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:35.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:35.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:35.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:35.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:35.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2026-02-22T20:18:35.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:35.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:35.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_19 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:35.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:35.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:35.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:35.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:35.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_19 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:35.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:35.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:35.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:35.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:35.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:35.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_19 2026-02-22T20:18:35.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_19 '' 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_19.status 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_19.status 2026-02-22T20:18:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:35.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:35.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2026-02-22T20:18:35.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:35.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:18:35.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:18:35.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:35.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242]]], state=connected] 2026-02-22T20:18:35.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242], [object_number=29, tag_tid=1, entry_tid=33241], [object_number=28, tag_tid=1, entry_tid=33240]]], state=connected] 2026-02-22T20:18:35.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_19.status 2026-02-22T20:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_19.status 2026-02-22T20:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2026-02-22T20:18:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:35.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:18:35.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:18:35.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:35.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242]]], state=connected] 2026-02-22T20:18:35.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242], [object_number=29, tag_tid=1, entry_tid=33241], [object_number=28, tag_tid=1, entry_tid=33240]]], state=connected] 2026-02-22T20:18:35.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=31, tag_tid=1, entry_tid=33243' 2026-02-22T20:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=31, tag_tid=1, entry_tid=33243' 2026-02-22T20:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=31, tag_tid=1, entry_tid=33243' '!=' '' 2026-02-22T20:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=31, tag_tid=1, entry_tid=33243' = '' 2026-02-22T20:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=31, tag_tid=1, entry_tid=33243' 2026-02-22T20:18:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:18:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:35.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:35.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:18:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=31, tag_tid=1, entry_tid=33243' 2026-02-22T20:18:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:35.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:35.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:18:35.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:18:35.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:35.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:18:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=31, tag_tid=1, entry_tid=33243' 2026-02-22T20:18:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:35.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=33243 2026-02-22T20:18:35.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 33243 2026-02-22T20:18:35.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:35.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:35.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:35.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_19 2026-02-22T20:18:35.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:35.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:35.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:35.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:35.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_19 ']' 2026-02-22T20:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_19' 2026-02-22T20:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2026-02-22T20:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:35.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2026-02-22T20:18:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:35.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:35.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:35.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:35.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:35.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2026-02-22T20:18:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:35.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:35.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2026-02-22T20:18:35.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:35.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:35.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:35.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:35.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:36.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2026-02-22T20:18:36.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:36.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:36.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:36.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_19 2026-02-22T20:18:36.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:36.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:36.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:36.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:36.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:36.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:36.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:36.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_19 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:18:36.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:36.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_19 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_19 '' 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_19.status 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_19.status 2026-02-22T20:18:36.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2026-02-22T20:18:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:18:36.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:18:36.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:36.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242]]], state=connected] 2026-02-22T20:18:36.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242], [object_number=29, tag_tid=1, entry_tid=33241], [object_number=28, tag_tid=1, entry_tid=33240]]], state=connected] 2026-02-22T20:18:36.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:18:36.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2026-02-22T20:18:36.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:36.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:36.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:36.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2026-02-22T20:18:36.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:36.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:36.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:36.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:36.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_19.status 2026-02-22T20:18:36.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_19.status 2026-02-22T20:18:36.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:36.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2026-02-22T20:18:36.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:18:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:18:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242]]], state=connected] 2026-02-22T20:18:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242], [object_number=29, tag_tid=1, entry_tid=33241], [object_number=28, tag_tid=1, entry_tid=33240]]], state=connected] 2026-02-22T20:18:36.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=31, tag_tid=1, entry_tid=33243' 2026-02-22T20:18:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=31, tag_tid=1, entry_tid=33243' 2026-02-22T20:18:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=31, tag_tid=1, entry_tid=33243' '!=' 'object_number=31, tag_tid=1, entry_tid=33243' 2026-02-22T20:18:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:36.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:18:36.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_19 2026-02-22T20:18:36.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:36.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:36.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:36.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:36.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_19 ']' 2026-02-22T20:18:36.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_19' 2026-02-22T20:18:36.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2026-02-22T20:18:36.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:36.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:36.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:36.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:36.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:36.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:36.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:36.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2026-02-22T20:18:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:36.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:36.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:36.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:36.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:36.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:36.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:36.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:36.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:36.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2026-02-22T20:18:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:36.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:36.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:36.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:36.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:36.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2026-02-22T20:18:36.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:36.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_19 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:36.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:36.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:36.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:18:36.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:36.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_19 2026-02-22T20:18:36.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:18:36.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:36.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:36.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:36.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:36.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:36.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_19 2026-02-22T20:18:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_19 2026-02-22T20:18:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_19 '' 2026-02-22T20:18:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_19.status 2026-02-22T20:18:36.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_19.status 2026-02-22T20:18:36.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:36.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2026-02-22T20:18:36.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242]]], state=connected] 2026-02-22T20:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242]]], state=connected] 2026-02-22T20:18:36.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:18:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2026-02-22T20:18:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2026-02-22T20:18:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:18:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:36.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_19.status 2026-02-22T20:18:36.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_19.status 2026-02-22T20:18:36.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:36.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:36.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2026-02-22T20:18:36.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:37.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:18:37.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:18:37.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:37.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242]]], state=connected] 2026-02-22T20:18:37.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=33243], [object_number=30, tag_tid=1, entry_tid=33242]]], state=connected] 2026-02-22T20:18:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.status 2026-02-22T20:18:37.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:18:37.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:18:37.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:37.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2026-02-22T20:18:37.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:37.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:18:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:38.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2026-02-22T20:18:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_19': 2026-02-22T20:18:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:18:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:18:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:18:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 117118d2927e 2026-02-22T20:18:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.117118d2927e 2026-02-22T20:18:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:18:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:09:56 2026 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:09:56 2026 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:11:02 2026 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 117118d2927e 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 638dc1d3-4b98-466a-9f37-5982b172fee8 2026-02-22T20:18:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:18:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_19 snap 2026-02-22T20:18:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:18:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:18:38.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.export 2026-02-22T20:18:38.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_19.export 2026-02-22T20:18:38.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_19.export 2026-02-22T20:18:38.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.export 2026-02-22T20:18:38.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_19.export 2026-02-22T20:18:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:38.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_19.export 2026-02-22T20:18:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_19.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_19.export 2026-02-22T20:18:38.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:18:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_20 2026-02-22T20:18:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_20 snap 2026-02-22T20:18:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:18:40.423 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2026-02-22T20:18:40.424 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:18:40.424 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2026-02-22T20:18:40.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2026-02-22T20:18:40.424 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2026-02-22T20:18:40.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2026-02-22T20:18:40.427 INFO:tasks.rbd_mirror_thrash:waiting for 103.88398052294121 secs before thrashing 2026-02-22T20:18:40.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2026-02-22T20:18:40.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:40.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:40.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:40.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2026-02-22T20:18:40.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:40.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2026-02-22T20:18:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:18:41.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:41.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:41.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:41.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:41.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2026-02-22T20:18:41.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2026-02-22T20:18:41.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:18:41.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:41.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:41.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:41.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:41.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:41.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:41.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:18:41.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:18:41.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:41.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:41.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2026-02-22T20:18:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:18:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:18:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:18:41.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:18:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_20 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:18:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:18:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_20 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_20 ']' 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_20' 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:42.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:42.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:42.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:42.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:42.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2026-02-22T20:18:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:42.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:42.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2026-02-22T20:18:42.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:42.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:42.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:42.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:42.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:42.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:42.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:42.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:42.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:42.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_20 2026-02-22T20:18:42.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:42.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:42.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:42.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '' 2026-02-22T20:18:42.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:42.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:42.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:42.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:42.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:42.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:42.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:42.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:42.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:42.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:42.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:42.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:42.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:42.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2026-02-22T20:18:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2026-02-22T20:18:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:42.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:42.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:42.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:42.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:42.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:42.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:42.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=16, tag_tid=1, entry_tid=19324' '!=' '' 2026-02-22T20:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=16, tag_tid=1, entry_tid=19324' = '' 2026-02-22T20:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:42.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:18:42.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:42.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:42.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:42.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:18:42.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:42.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:42.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:42.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:18:42.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:42.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:42.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:42.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=19324 2026-02-22T20:18:42.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 19324 2026-02-22T20:18:42.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:42.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:42.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_20 2026-02-22T20:18:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_20 ']' 2026-02-22T20:18:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_20' 2026-02-22T20:18:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2026-02-22T20:18:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:42.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:42.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2026-02-22T20:18:42.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:42.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:42.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:42.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:42.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:42.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2026-02-22T20:18:42.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:42.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:42.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:42.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:42.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:42.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:42.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:42.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_20 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '' 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:42.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:42.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:42.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:42.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:42.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:42.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:42.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:42.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2026-02-22T20:18:42.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:42.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:42.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:42.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2026-02-22T20:18:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:42.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:42.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:42.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:42.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:42.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=16, tag_tid=1, entry_tid=19324' '!=' 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_20 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_20 ']' 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_20' 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:43.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:43.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:43.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2026-02-22T20:18:43.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:43.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:43.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:43.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_20 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '' 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:43.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:43.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:43.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:43.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:43.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:43.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:43.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:43.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:43.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:43.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:43.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:43.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:43.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:43.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=16, tag_tid=1, entry_tid=19324' '!=' 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:43.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:43.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_20 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_20 ']' 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_20' 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:44.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:44.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:44.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:44.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:44.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:44.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2026-02-22T20:18:44.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:44.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:44.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:44.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:44.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:44.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:44.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:44.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:44.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:44.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:44.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:44.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:44.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:44.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:44.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_20 2026-02-22T20:18:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '' 2026-02-22T20:18:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:44.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2026-02-22T20:18:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2026-02-22T20:18:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:44.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:44.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:44.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:44.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:44.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:44.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:44.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=16, tag_tid=1, entry_tid=19324' '!=' 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:18:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_20 2026-02-22T20:18:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_20 ']' 2026-02-22T20:18:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_20' 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2026-02-22T20:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:46.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:46.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:46.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:46.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:46.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:46.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:46.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:46.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:46.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:46.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:46.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2026-02-22T20:18:46.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:46.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:46.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:46.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2026-02-22T20:18:46.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:46.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:46.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:46.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:46.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:46.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:46.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:46.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:46.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:46.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:46.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:46.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_20 2026-02-22T20:18:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '' 2026-02-22T20:18:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:46.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:46.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:46.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:46.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:46.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:46.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:46.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:46.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:46.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=16, tag_tid=1, entry_tid=19324], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322], [object_number=17, tag_tid=1, entry_tid=19321]]], state=connected] 2026-02-22T20:18:46.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=16, tag_tid=1, entry_tid=19324' '!=' 'object_number=16, tag_tid=1, entry_tid=19324' 2026-02-22T20:18:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:18:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_20 2026-02-22T20:18:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_20 ']' 2026-02-22T20:18:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_20' 2026-02-22T20:18:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2026-02-22T20:18:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:48.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:48.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:48.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:48.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:48.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:48.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:48.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:48.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:48.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:48.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:48.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:48.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:48.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:48.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2026-02-22T20:18:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:48.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:48.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:48.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:48.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:48.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2026-02-22T20:18:48.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:48.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_20 2026-02-22T20:18:48.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:48.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:48.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:48.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '' 2026-02-22T20:18:48.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:48.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:48.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_20.status 2026-02-22T20:18:48.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_20.status 2026-02-22T20:18:48.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:48.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:48.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2026-02-22T20:18:48.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:48.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:18:48.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:18:48.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:48.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:48.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=19323], [object_number=18, tag_tid=1, entry_tid=19322]]], state=connected] 2026-02-22T20:18:48.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.status 2026-02-22T20:18:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:18:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2026-02-22T20:18:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:48.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:18:48.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:49.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_20': 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e31f25d5ed 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e31f25d5ed 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:10:07 2026 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:10:07 2026 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:10:07 2026 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e31f25d5ed 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 5906cf9d-5548-4aab-a5b0-83a254416894 2026-02-22T20:18:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_20 snap 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.export 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_20.export 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_20.export 2026-02-22T20:18:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.export 2026-02-22T20:18:49.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:49.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_20.export 2026-02-22T20:18:50.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:18:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_20.export 2026-02-22T20:18:50.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_20.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_20.export 2026-02-22T20:18:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:18:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_21 2026-02-22T20:18:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_21 snap 2026-02-22T20:18:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:18:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:18:52.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2026-02-22T20:18:52.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:52.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2026-02-22T20:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:18:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2026-02-22T20:18:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:18:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:18:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:18:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2026-02-22T20:18:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:18:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:18:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:53.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:18:53.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:18:53.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:53.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:18:53.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:53.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:18:53.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:53.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2026-02-22T20:18:53.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:18:53.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:53.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:53.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:53.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:53.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:53.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2026-02-22T20:18:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:18:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:53.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:53.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2026-02-22T20:18:53.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:18:53.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:53.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:53.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:18:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:18:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:53.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:53.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:53.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:53.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2026-02-22T20:18:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:18:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:53.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:53.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:53.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:53.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2026-02-22T20:18:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:18:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_21 2026-02-22T20:18:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:18:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:54.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:18:54.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:18:54.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:18:54.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:18:54.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:54.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_21 2026-02-22T20:18:54.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:18:54.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:18:54.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:18:54.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:18:54.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:18:54.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:18:54.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:18:54.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:54.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_21 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:54.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:54.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:18:54.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:18:54.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:54.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:54.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_21 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_21 ']' 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_21' 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:54.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:54.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:54.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:54.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:54.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:54.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:54.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:54.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:54.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2026-02-22T20:18:54.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:54.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:54.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:54.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:54.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:54.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:54.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:54.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:54.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:54.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:54.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:54.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_21 2026-02-22T20:18:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:54.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:54.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:54.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:54.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:54.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:54.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:54.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:54.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:54.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:54.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_21 2026-02-22T20:18:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:54.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:54.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:54.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:54.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:54.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:54.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_21 2026-02-22T20:18:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '' 2026-02-22T20:18:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:54.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:54.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:54.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:54.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:54.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:54.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:54.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:54.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:54.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:54.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:54.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:54.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:54.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:54.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:54.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:54.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:54.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:54.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:54.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:54.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:54.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:54.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=35101' '!=' '' 2026-02-22T20:18:54.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:18:54.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=35101' = '' 2026-02-22T20:18:54.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:54.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:54.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:54.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:54.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:18:54.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:54.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:18:54.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:54.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:18:54.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:54.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:54.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:18:54.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:18:54.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:18:54.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35101 2026-02-22T20:18:54.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 35101 2026-02-22T20:18:54.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:18:54.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:54.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:18:54.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_21 2026-02-22T20:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_21 ']' 2026-02-22T20:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_21' 2026-02-22T20:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2026-02-22T20:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:54.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:54.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:54.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2026-02-22T20:18:54.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:54.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:54.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:54.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:54.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:54.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:54.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:54.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:54.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:54.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2026-02-22T20:18:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:54.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:54.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:54.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:54.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:54.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:55.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:55.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:55.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:55.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:55.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_21 2026-02-22T20:18:55.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:55.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:55.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:55.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:55.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:55.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:55.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:55.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:55.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:55.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_21 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:55.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:55.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:55.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:55.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:55.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_21 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '' 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:55.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:55.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:55.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2026-02-22T20:18:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2026-02-22T20:18:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:55.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:55.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:55.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:55.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:55.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:55.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:55.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:55.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:55.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=35101' '!=' 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:18:55.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_21 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_21 ']' 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_21' 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:55.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:55.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:55.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:55.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:55.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:55.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:55.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:55.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:55.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:55.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:55.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:55.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:55.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:55.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:55.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:55.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:55.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_21 2026-02-22T20:18:55.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:55.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:55.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:55.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:55.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:55.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:55.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:55.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_21 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:55.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:55.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:55.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:55.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_21 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '' 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:55.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:55.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:55.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:55.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:55.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:55.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:55.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:55.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:55.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:55.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:55.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:55.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:55.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2026-02-22T20:18:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2026-02-22T20:18:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:56.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:56.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:56.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:56.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:56.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:56.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:56.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:56.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=35101' '!=' 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_21 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_21 ']' 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_21' 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:56.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:56.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:56.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:56.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2026-02-22T20:18:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:56.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:56.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:56.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_21 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:56.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:56.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:56.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:56.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:56.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:56.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:56.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:56.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:56.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:56.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:57.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:57.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:57.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_21 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:57.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:57.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:57.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:57.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:57.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:57.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:57.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_21 2026-02-22T20:18:57.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:57.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:57.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '' 2026-02-22T20:18:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:57.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:57.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:57.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:57.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:57.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:57.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:57.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2026-02-22T20:18:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2026-02-22T20:18:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:57.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:57.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:57.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:57.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:57.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:57.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:57.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:57.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:57.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:57.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:57.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:57.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=35101' '!=' 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:57.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:57.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_21 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_21 ']' 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_21' 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:18:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:18:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:18:58.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:18:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:18:58.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:18:58.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:58.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:58.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:18:58.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:58.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:58.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:58.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2026-02-22T20:18:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:18:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:18:58.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:58.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:58.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:18:58.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:58.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:58.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_21 2026-02-22T20:18:58.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:18:58.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:58.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:58.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:18:58.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:18:58.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:18:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:18:58.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:58.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:58.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:18:58.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:59.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:18:59.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:18:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_21 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:18:59.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:18:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:18:59.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:59.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:59.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:18:59.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_21 2026-02-22T20:18:59.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:18:59.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_21 2026-02-22T20:18:59.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:59.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:59.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:59.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '' 2026-02-22T20:18:59.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:59.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:59.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:18:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:59.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:59.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:18:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:18:59.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:18:59.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:18:59.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:18:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:59.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:18:59.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:18:59.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:18:59.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:59.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=1, entry_tid=35101], [object_number=32, tag_tid=1, entry_tid=35100], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:18:59.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:18:59.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:59.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:59.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=33, tag_tid=1, entry_tid=35101' '!=' 'object_number=33, tag_tid=1, entry_tid=35101' 2026-02-22T20:18:59.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:18:59.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:19:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_21 2026-02-22T20:19:01.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_21 ']' 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_21' 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:01.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:01.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:01.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:01.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:01.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2026-02-22T20:19:01.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2026-02-22T20:19:01.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:19:01.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:01.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:01.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:01.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2026-02-22T20:19:01.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:01.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:01.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:01.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:01.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:01.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:01.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:01.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2026-02-22T20:19:01.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:19:01.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:01.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_21 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:01.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:01.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:01.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:01.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:01.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:01.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_21 2026-02-22T20:19:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2026-02-22T20:19:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:01.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_21 2026-02-22T20:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:19:01.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:19:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:19:01.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:19:01.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:19:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:19:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:19:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_21 2026-02-22T20:19:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_21 2026-02-22T20:19:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:19:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '' 2026-02-22T20:19:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:19:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:19:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:19:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:19:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:19:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:19:01.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:19:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:19:01.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2026-02-22T20:19:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:19:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2026-02-22T20:19:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:19:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_21.status 2026-02-22T20:19:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_21.status 2026-02-22T20:19:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:19:01.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2026-02-22T20:19:01.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:19:01.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:01.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:01.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:01.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:19:01.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35099], [object_number=34, tag_tid=1, entry_tid=35098]]], state=connected] 2026-02-22T20:19:01.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.status 2026-02-22T20:19:01.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:19:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:02.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2026-02-22T20:19:02.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_21': 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aa8037c64c 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aa8037c64c 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:19:02.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:10:14 2026 2026-02-22T20:19:02.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:10:14 2026 2026-02-22T20:19:02.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:11:15 2026 2026-02-22T20:19:02.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:19:02.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aa8037c64c 2026-02-22T20:19:02.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:19:02.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:19:02.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 7640030b-1c0d-44de-bdeb-0c8ca381debc 2026-02-22T20:19:02.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:19:02.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_21 snap 2026-02-22T20:19:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:19:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:19:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.export 2026-02-22T20:19:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_21.export 2026-02-22T20:19:02.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_21.export 2026-02-22T20:19:02.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.export 2026-02-22T20:19:02.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_21.export 2026-02-22T20:19:02.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:02.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_21.export 2026-02-22T20:19:03.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_21.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_21.export 2026-02-22T20:19:03.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:03.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:19:03.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_22 2026-02-22T20:19:03.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_22 snap 2026-02-22T20:19:03.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:03.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:03.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:19:03.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:03.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2026-02-22T20:19:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:19:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:19:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:19:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:19:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2026-02-22T20:19:04.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:19:04.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:19:04.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:04.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:04.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:19:04.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:19:04.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:04.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:19:04.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:04.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:04.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:19:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:19:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:04.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:04.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2026-02-22T20:19:04.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2026-02-22T20:19:04.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:19:04.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:19:04.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:19:04.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:04.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2026-02-22T20:19:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:19:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:19:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:19:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:19:05.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_22 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_22 2026-02-22T20:19:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_22 ']' 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_22' 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:05.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:05.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:05.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:05.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:05.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2026-02-22T20:19:05.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2026-02-22T20:19:05.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:05.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2026-02-22T20:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2026-02-22T20:19:05.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:05.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_22 2026-02-22T20:19:05.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:05.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_22 2026-02-22T20:19:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_22 '' 2026-02-22T20:19:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_22 2026-02-22T20:19:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_22.status 2026-02-22T20:19:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_22.status 2026-02-22T20:19:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:05.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.status 2026-02-22T20:19:05.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2026-02-22T20:19:05.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.status 2026-02-22T20:19:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12354], [object_number=11, tag_tid=1, entry_tid=12351]]], state=connected] 2026-02-22T20:19:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12354], [object_number=11, tag_tid=1, entry_tid=12351]]], state=connected] 2026-02-22T20:19:05.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.status 2026-02-22T20:19:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:19:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2026-02-22T20:19:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_22 2026-02-22T20:19:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2026-02-22T20:19:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_22 2026-02-22T20:19:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_22.status 2026-02-22T20:19:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_22.status 2026-02-22T20:19:05.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:05.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.status 2026-02-22T20:19:05.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2026-02-22T20:19:05.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.status 2026-02-22T20:19:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:05.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12354], [object_number=11, tag_tid=1, entry_tid=12351]]], state=connected] 2026-02-22T20:19:05.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12354], [object_number=11, tag_tid=1, entry_tid=12351]]], state=connected] 2026-02-22T20:19:05.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.status 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:19:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:06.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2026-02-22T20:19:06.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_22': 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 112792a568bf 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.112792a568bf 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:10:25 2026 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:10:25 2026 2026-02-22T20:19:06.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:16:47 2026 2026-02-22T20:19:06.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:19:06.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 112792a568bf 2026-02-22T20:19:06.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:19:06.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:19:06.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 515162a4-1d39-49ab-bdbb-cc78cae19b2e 2026-02-22T20:19:06.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:19:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_22 snap 2026-02-22T20:19:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:19:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:19:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.export 2026-02-22T20:19:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_22.export 2026-02-22T20:19:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_22.export 2026-02-22T20:19:06.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.export 2026-02-22T20:19:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_22.export 2026-02-22T20:19:06.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_22.export 2026-02-22T20:19:06.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_22.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_22.export 2026-02-22T20:19:06.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:06.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:19:06.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_23 2026-02-22T20:19:06.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_23 snap 2026-02-22T20:19:06.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:06.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:06.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:06.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:06.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:19:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2026-02-22T20:19:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2026-02-22T20:19:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:19:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:07.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:08.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2026-02-22T20:19:08.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:08.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:08.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:08.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:19:08.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:19:08.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:19:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2026-02-22T20:19:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:19:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:19:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:19:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:19:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:19:08.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:08.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:19:08.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:08.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2026-02-22T20:19:08.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:19:08.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:08.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:08.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:08.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:19:08.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:19:08.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:19:08.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2026-02-22T20:19:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:19:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:19:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:19:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:19:08.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:19:08.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:19:08.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:19:08.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:08.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2026-02-22T20:19:08.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:08.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_23 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:08.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:08.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_23 2026-02-22T20:19:09.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_23 ']' 2026-02-22T20:19:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_23' 2026-02-22T20:19:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2026-02-22T20:19:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:09.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:09.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:09.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:09.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:09.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:09.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:09.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:09.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2026-02-22T20:19:09.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:09.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:09.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:09.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:09.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:09.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2026-02-22T20:19:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_23 2026-02-22T20:19:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '' 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:09.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:09.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:09.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:09.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:09.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:09.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:09.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:09.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:09.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:09.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:09.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:09.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:09.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:09.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=3, entry_tid=1' -a 'object_number=17, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=1' '!=' '' 2026-02-22T20:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=1' = '' 2026-02-22T20:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T20:19:09.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:09.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:09.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:09.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=2 2026-02-22T20:19:09.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:09.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:09.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:09.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:19:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=1 2026-02-22T20:19:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 2 -a 1 -le 1 2026-02-22T20:19:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:09.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_23 2026-02-22T20:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_23 ']' 2026-02-22T20:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_23' 2026-02-22T20:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2026-02-22T20:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:09.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:09.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:09.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:09.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:09.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:09.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2026-02-22T20:19:09.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_23 2026-02-22T20:19:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '' 2026-02-22T20:19:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:09.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:09.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:09.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:09.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:09.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:09.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:09.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:09.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:09.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:09.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:09.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2026-02-22T20:19:09.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:09.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:09.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:09.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2026-02-22T20:19:09.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:09.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:09.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:09.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:09.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:09.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:09.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:09.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:09.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:09.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=3, entry_tid=1' -a 'object_number=17, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:09.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=1' '!=' 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:09.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:09.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_23 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_23 ']' 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_23' 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:10.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:10.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:10.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:10.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:10.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:10.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_23 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '' 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:10.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:10.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:10.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:10.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:10.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:10.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:10.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:10.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:10.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:10.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=3, entry_tid=1' -a 'object_number=17, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:10.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=1' '!=' 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:10.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:10.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_23 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_23 ']' 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_23' 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:11.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:11.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2026-02-22T20:19:11.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:11.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:11.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:11.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:11.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:11.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:11.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:11.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:11.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:11.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:11.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2026-02-22T20:19:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_23 2026-02-22T20:19:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '' 2026-02-22T20:19:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:11.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:11.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:11.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:11.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2026-02-22T20:19:11.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:11.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2026-02-22T20:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:11.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:11.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:11.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:11.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:11.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:11.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:11.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:11.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:11.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:11.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:11.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=3, entry_tid=1' -a 'object_number=17, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:11.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=1' '!=' 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:11.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:11.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:19:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_23 2026-02-22T20:19:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_23 ']' 2026-02-22T20:19:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_23' 2026-02-22T20:19:12.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2026-02-22T20:19:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:12.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2026-02-22T20:19:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:12.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:12.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:12.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2026-02-22T20:19:12.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_23 2026-02-22T20:19:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '' 2026-02-22T20:19:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:12.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:12.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:12.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:12.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:12.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:12.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:12.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:12.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:12.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:12.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:12.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:12.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=3, entry_tid=1' -a 'object_number=17, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:12.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=1' '!=' 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:12.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:12.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:19:14.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_23 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_23 ']' 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_23' 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:14.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:14.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2026-02-22T20:19:14.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:14.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:14.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:14.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:14.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:14.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2026-02-22T20:19:15.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:15.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_23 2026-02-22T20:19:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '' 2026-02-22T20:19:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:15.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:15.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:15.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:15.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:15.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:15.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:15.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:15.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:15.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:15.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=3, entry_tid=1' -a 'object_number=17, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=1' '!=' 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:19:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:19:17.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_23 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_23 ']' 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_23' 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:17.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:17.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:17.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:17.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:17.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2026-02-22T20:19:17.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_23 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '' 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:17.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:17.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:17.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:17.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:17.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:17.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:17.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:17.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:17.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2026-02-22T20:19:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2026-02-22T20:19:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:19:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:17.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_23.status 2026-02-22T20:19:17.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_23.status 2026-02-22T20:19:17.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:17.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:17.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2026-02-22T20:19:17.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:17.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:19:17.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:19:17.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:17.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:17.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=20695], [object_number=18, tag_tid=1, entry_tid=20694]]], state=connected] 2026-02-22T20:19:17.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.status 2026-02-22T20:19:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=3, entry_tid=1' -a 'object_number=17, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=3, entry_tid=1' 2026-02-22T20:19:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2026-02-22T20:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2026-02-22T20:19:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_23': 2026-02-22T20:19:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:19:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:19:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1171b1c534aa 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1171b1c534aa 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:10:30 2026 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:10:30 2026 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:13:08 2026 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1171b1c534aa 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 38bc1dca-7653-4af8-82a2-a90b045ed9d3 2026-02-22T20:19:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:19:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_23 snap 2026-02-22T20:19:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:19:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:19:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.export 2026-02-22T20:19:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_23.export 2026-02-22T20:19:18.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_23.export 2026-02-22T20:19:18.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.export 2026-02-22T20:19:18.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:18.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_23.export 2026-02-22T20:19:18.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:18.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_23.export 2026-02-22T20:19:18.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_23.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_23.export 2026-02-22T20:19:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:19:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_24 2026-02-22T20:19:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_24 snap 2026-02-22T20:19:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:19:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2026-02-22T20:19:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2026-02-22T20:19:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:19:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2026-02-22T20:19:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:21.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:19:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:21.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:19:21.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:21.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:19:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:21.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:21.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:21.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:21.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:21.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:21.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:21.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:19:21.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:19:21.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:21.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:21.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:21.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:21.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2026-02-22T20:19:21.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2026-02-22T20:19:21.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:19:21.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:21.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:21.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2026-02-22T20:19:21.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:19:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:19:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:21.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:21.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:21.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2026-02-22T20:19:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:19:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:19:22.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:19:22.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:19:22.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:19:22.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_24 2026-02-22T20:19:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:19:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:19:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:22.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_24 2026-02-22T20:19:22.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:22.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:22.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:22.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:22.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_24 ']' 2026-02-22T20:19:22.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_24' 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:22.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:22.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2026-02-22T20:19:22.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:22.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:22.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:22.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:22.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:22.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:22.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:22.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:22.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:22.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:22.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2026-02-22T20:19:22.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2026-02-22T20:19:22.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:22.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:22.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:22.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2026-02-22T20:19:22.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:22.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:22.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:22.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:22.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:22.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:22.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:22.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:22.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:22.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:22.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:22.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:22.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_24 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_24 '' 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:22.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_24.status 2026-02-22T20:19:22.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_24.status 2026-02-22T20:19:22.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:22.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2026-02-22T20:19:22.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:22.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:22.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:22.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:22.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734], [object_number=33, tag_tid=1, entry_tid=35733], [object_number=32, tag_tid=1, entry_tid=35732]]], state=connected] 2026-02-22T20:19:22.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:22.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_24.status 2026-02-22T20:19:22.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_24.status 2026-02-22T20:19:22.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:22.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2026-02-22T20:19:22.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734], [object_number=33, tag_tid=1, entry_tid=35733], [object_number=32, tag_tid=1, entry_tid=35732]]], state=connected] 2026-02-22T20:19:22.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=35, tag_tid=1, entry_tid=35735' '!=' '' 2026-02-22T20:19:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:19:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=35, tag_tid=1, entry_tid=35735' = '' 2026-02-22T20:19:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:22.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:22.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:19:22.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:22.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:22.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:19:22.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:22.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:22.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:22.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:19:22.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:22.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:22.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:22.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=35735 2026-02-22T20:19:22.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 35735 2026-02-22T20:19:22.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:22.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:22.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_24 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_24 ']' 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_24' 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2026-02-22T20:19:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:22.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:22.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:22.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:22.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2026-02-22T20:19:22.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2026-02-22T20:19:22.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:22.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:22.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:22.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:22.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:22.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2026-02-22T20:19:22.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_24 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_24 '' 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_24.status 2026-02-22T20:19:22.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_24.status 2026-02-22T20:19:22.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2026-02-22T20:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:22.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734], [object_number=33, tag_tid=1, entry_tid=35733], [object_number=32, tag_tid=1, entry_tid=35732]]], state=connected] 2026-02-22T20:19:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_24.status 2026-02-22T20:19:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_24.status 2026-02-22T20:19:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2026-02-22T20:19:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:22.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:22.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:22.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:22.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734], [object_number=33, tag_tid=1, entry_tid=35733], [object_number=32, tag_tid=1, entry_tid=35732]]], state=connected] 2026-02-22T20:19:22.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:22.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:22.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:22.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=35, tag_tid=1, entry_tid=35735' '!=' 'object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:22.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:22.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:19:23.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_24 2026-02-22T20:19:23.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:23.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:23.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:23.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:23.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_24 ']' 2026-02-22T20:19:23.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_24' 2026-02-22T20:19:23.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2026-02-22T20:19:23.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:23.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:23.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:23.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:23.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:23.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2026-02-22T20:19:23.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:23.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:23.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:23.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:23.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:23.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:23.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:23.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:23.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:23.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:23.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:23.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:23.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:23.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:23.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:23.303 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:23.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:23.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:23.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:23.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2026-02-22T20:19:23.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2026-02-22T20:19:23.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:23.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:23.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2026-02-22T20:19:23.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_24 2026-02-22T20:19:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_24 '' 2026-02-22T20:19:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:23.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_24.status 2026-02-22T20:19:23.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_24.status 2026-02-22T20:19:23.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:23.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:23.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2026-02-22T20:19:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:23.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:23.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:23.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:23.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:23.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734], [object_number=33, tag_tid=1, entry_tid=35733], [object_number=32, tag_tid=1, entry_tid=35732]]], state=connected] 2026-02-22T20:19:23.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:23.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:23.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2026-02-22T20:19:23.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:23.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:23.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:23.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2026-02-22T20:19:23.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:23.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:23.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:23.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:23.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_24.status 2026-02-22T20:19:23.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_24.status 2026-02-22T20:19:23.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:23.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:23.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2026-02-22T20:19:23.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:23.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:23.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:23.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:23.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:23.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734], [object_number=33, tag_tid=1, entry_tid=35733], [object_number=32, tag_tid=1, entry_tid=35732]]], state=connected] 2026-02-22T20:19:23.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:23.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:23.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:23.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=35, tag_tid=1, entry_tid=35735' '!=' 'object_number=35, tag_tid=1, entry_tid=35735' 2026-02-22T20:19:23.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:23.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_24 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_24 ']' 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_24' 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:24.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:24.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:24.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2026-02-22T20:19:24.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2026-02-22T20:19:24.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:24.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:24.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:24.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:24.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:24.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2026-02-22T20:19:24.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_24 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_24 '' 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_24.status 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_24.status 2026-02-22T20:19:24.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:24.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:24.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2026-02-22T20:19:24.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:24.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:24.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:24.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:24.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2026-02-22T20:19:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2026-02-22T20:19:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:19:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_24.status 2026-02-22T20:19:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_24.status 2026-02-22T20:19:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:24.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:24.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2026-02-22T20:19:24.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:24.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:19:24.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:19:24.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:24.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:24.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=35735], [object_number=34, tag_tid=1, entry_tid=35734]]], state=connected] 2026-02-22T20:19:24.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.status 2026-02-22T20:19:24.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:19:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:25.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2026-02-22T20:19:25.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_24': 2026-02-22T20:19:25.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:19:25.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e3e570d1e2 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e3e570d1e2 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:10:37 2026 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:10:37 2026 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:11:38 2026 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:19:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e3e570d1e2 2026-02-22T20:19:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:19:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:19:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 148f084b-aba8-4997-a432-c6dcf95539c7 2026-02-22T20:19:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:19:25.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:25.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_24 snap 2026-02-22T20:19:25.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:25.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:19:25.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:25.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:19:25.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.export 2026-02-22T20:19:25.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_24.export 2026-02-22T20:19:25.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_24.export 2026-02-22T20:19:25.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.export 2026-02-22T20:19:25.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:25.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_24.export 2026-02-22T20:19:25.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:25.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_24.export 2026-02-22T20:19:25.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_24.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_24.export 2026-02-22T20:19:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:19:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_25 2026-02-22T20:19:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_25 snap 2026-02-22T20:19:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:26.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:26.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:27.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:28.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2026-02-22T20:19:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:19:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:19:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:19:28.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2026-02-22T20:19:28.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:19:28.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:19:28.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:28.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:28.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:19:28.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:19:28.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:28.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:19:28.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:28.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:19:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:19:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:19:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2026-02-22T20:19:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:19:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2026-02-22T20:19:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:19:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:19:28.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:19:28.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:28.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:28.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:28.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:28.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2026-02-22T20:19:29.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:29.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:19:29.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:29.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:29.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2026-02-22T20:19:29.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:19:29.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:29.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:29.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:19:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:19:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:29.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:29.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:29.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:29.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2026-02-22T20:19:29.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:19:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:19:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:19:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:19:29.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:19:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:19:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:19:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2026-02-22T20:19:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_25 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:29.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:29.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_25 2026-02-22T20:19:29.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:29.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_25 ']' 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_25' 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:29.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:29.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:29.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:29.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:29.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:29.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:29.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:29.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:29.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:29.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2026-02-22T20:19:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:29.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:29.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:29.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:29.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_25 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:29.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:29.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:29.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_25 2026-02-22T20:19:29.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:29.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:29.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:29.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '' 2026-02-22T20:19:29.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:29.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:29.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:29.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:29.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:29.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:29.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:29.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:29.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:29.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:29.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:29.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:29.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:29.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:29.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13929], [object_number=12, tag_tid=1, entry_tid=13928], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:29.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:29.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:19:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2026-02-22T20:19:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2026-02-22T20:19:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:29.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:29.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:29.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:29.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:29.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13929], [object_number=12, tag_tid=1, entry_tid=13928], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:29.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=13929' '!=' '' 2026-02-22T20:19:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:19:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=13929' = '' 2026-02-22T20:19:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:29.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:19:29.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:29.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:29.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:29.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:19:29.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:19:29.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:29.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:29.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:19:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=13929 2026-02-22T20:19:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 13929 2026-02-22T20:19:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_25 2026-02-22T20:19:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:29.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_25 ']' 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_25' 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:29.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:29.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:29.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:29.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:29.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2026-02-22T20:19:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:29.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:29.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:29.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:29.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:29.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:29.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:29.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:29.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:29.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:29.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:29.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:29.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:29.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2026-02-22T20:19:29.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:29.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:29.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:29.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:29.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:29.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:29.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:29.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:29.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:29.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_25 2026-02-22T20:19:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:29.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:29.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:29.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:29.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:29.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:29.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:29.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:29.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:29.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:29.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:29.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:29.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:29.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:29.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:29.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:30.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:30.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_25 2026-02-22T20:19:30.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:30.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:30.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '' 2026-02-22T20:19:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:30.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:30.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:30.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:30.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:30.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:30.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13929], [object_number=12, tag_tid=1, entry_tid=13928], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:30.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:30.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:30.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:30.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:30.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:30.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:30.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:30.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:30.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:30.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13929], [object_number=12, tag_tid=1, entry_tid=13928], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:30.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:30.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:30.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=13929' '!=' 'object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:30.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:30.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:19:30.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_25 2026-02-22T20:19:30.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:30.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:30.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_25 ']' 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_25' 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:30.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:30.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:30.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:30.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:30.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:30.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:30.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:30.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:30.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:30.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:30.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:30.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:30.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:30.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:30.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:30.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:30.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:30.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:30.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:30.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:30.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:30.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:30.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:30.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_25 2026-02-22T20:19:30.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:30.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:30.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:30.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:30.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:30.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:30.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_25 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '' 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:30.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:30.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:30.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:30.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:30.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:30.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:30.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:30.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13929], [object_number=12, tag_tid=1, entry_tid=13928], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:30.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:19:30.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2026-02-22T20:19:30.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:30.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:30.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:30.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2026-02-22T20:19:30.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:30.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:30.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:30.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:30.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:30.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:30.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:30.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:30.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:30.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:30.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13929], [object_number=12, tag_tid=1, entry_tid=13928], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:30.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=13929' '!=' 'object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:30.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:30.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:19:31.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_25 2026-02-22T20:19:31.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:31.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:31.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:31.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:31.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_25 ']' 2026-02-22T20:19:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_25' 2026-02-22T20:19:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2026-02-22T20:19:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:31.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:31.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:31.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:31.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:31.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2026-02-22T20:19:31.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:31.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:31.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:31.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:31.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:31.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:31.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:31.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:31.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:31.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2026-02-22T20:19:31.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:31.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:31.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:31.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:31.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:31.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:31.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:31.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:31.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:31.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_25 2026-02-22T20:19:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:31.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:31.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:31.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:31.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:31.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_25 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '' 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:31.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:31.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13929], [object_number=12, tag_tid=1, entry_tid=13928], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:31.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:31.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:19:31.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2026-02-22T20:19:31.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:31.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:31.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:31.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2026-02-22T20:19:31.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:31.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:31.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:31.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:31.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:31.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:31.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13929], [object_number=12, tag_tid=1, entry_tid=13928], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:31.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:31.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:31.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:31.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=13929' '!=' 'object_number=13, tag_tid=1, entry_tid=13929' 2026-02-22T20:19:31.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:31.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:19:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_25 2026-02-22T20:19:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_25 ']' 2026-02-22T20:19:33.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_25' 2026-02-22T20:19:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2026-02-22T20:19:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:33.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:33.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:33.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:33.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:33.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:33.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:33.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:33.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:33.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:33.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:33.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:33.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:33.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:33.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:33.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:33.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:33.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:33.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:33.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:33.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:33.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:33.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:33.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2026-02-22T20:19:33.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_25 2026-02-22T20:19:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:33.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:33.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:33.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:33.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:33.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_25 2026-02-22T20:19:33.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_25 2026-02-22T20:19:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '' 2026-02-22T20:19:33.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:33.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:33.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:33.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:33.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:33.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:33.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:33.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:33.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:33.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:33.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:19:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2026-02-22T20:19:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2026-02-22T20:19:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:19:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:33.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_25.status 2026-02-22T20:19:33.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_25.status 2026-02-22T20:19:33.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2026-02-22T20:19:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:19:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:19:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13927], [object_number=14, tag_tid=1, entry_tid=13926]]], state=connected] 2026-02-22T20:19:33.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.status 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:19:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2026-02-22T20:19:34.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_25': 2026-02-22T20:19:34.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:19:34.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:19:34.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:19:34.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11f8d56d9c84 2026-02-22T20:19:34.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11f8d56d9c84 2026-02-22T20:19:34.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:10:47 2026 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:10:47 2026 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:10:47 2026 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11f8d56d9c84 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 64115030-ef11-4dae-9a16-ebc4e7f2c446 2026-02-22T20:19:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:19:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_25 snap 2026-02-22T20:19:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:19:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:19:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.export 2026-02-22T20:19:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_25.export 2026-02-22T20:19:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_25.export 2026-02-22T20:19:34.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.export 2026-02-22T20:19:35.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:35.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_25.export 2026-02-22T20:19:35.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_25.export 2026-02-22T20:19:35.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_25.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_25.export 2026-02-22T20:19:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:19:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_26 2026-02-22T20:19:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_26 snap 2026-02-22T20:19:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:19:36.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2026-02-22T20:19:36.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:36.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:36.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:36.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2026-02-22T20:19:36.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:36.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:36.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:36.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:19:36.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:36.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:36.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:19:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:19:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:19:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:19:37.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:37.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:37.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:37.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:37.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:37.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2026-02-22T20:19:37.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:37.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:19:37.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:37.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:19:37.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:37.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:37.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2026-02-22T20:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:19:38.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2026-02-22T20:19:38.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:19:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:19:38.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:19:38.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:19:38.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:19:38.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_26 2026-02-22T20:19:38.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:38.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:38.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:38.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:38.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:19:38.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:19:38.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:38.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:38.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:38.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_26 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_26 ']' 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_26' 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:38.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:38.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:38.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:38.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:38.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:38.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:38.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:38.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:38.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:38.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:38.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:38.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2026-02-22T20:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:38.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:38.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:38.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:38.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:38.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:38.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:38.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:38.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_26 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:38.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:38.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:38.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:38.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:38.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:38.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:38.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_26 2026-02-22T20:19:38.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '' 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:38.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:38.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:38.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:38.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:38.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:38.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:38.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:38.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:38.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:38.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:38.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:38.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:38.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:38.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=11205' '!=' '' 2026-02-22T20:19:38.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:19:38.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=11205' = '' 2026-02-22T20:19:38.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:38.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:38.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:38.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:38.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:19:38.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:38.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:38.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:38.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:19:38.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:38.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:38.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:38.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:19:38.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:38.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:38.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=11205 2026-02-22T20:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 11205 2026-02-22T20:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:38.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:38.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_26 2026-02-22T20:19:38.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:38.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:38.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_26 ']' 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_26' 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:38.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2026-02-22T20:19:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:38.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:38.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:38.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:38.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:38.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:38.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:38.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2026-02-22T20:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:38.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:38.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:38.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:38.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:38.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:38.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:38.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:38.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:38.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:38.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:38.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:38.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:38.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:38.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_26 2026-02-22T20:19:38.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:38.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:38.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:38.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:38.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:38.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:38.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:38.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:38.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:38.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:38.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_26 2026-02-22T20:19:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '' 2026-02-22T20:19:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:39.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:39.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:39.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:39.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:39.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:39.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:39.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2026-02-22T20:19:39.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:39.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:39.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:39.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2026-02-22T20:19:39.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:39.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:39.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:39.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:39.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:39.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:39.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:39.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:39.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=11205' '!=' 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:19:39.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_26 2026-02-22T20:19:39.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:39.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:39.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:39.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:39.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_26 ']' 2026-02-22T20:19:39.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_26' 2026-02-22T20:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2026-02-22T20:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2026-02-22T20:19:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:39.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:39.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:39.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:39.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:39.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:39.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:39.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:39.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:39.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:39.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:39.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:39.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:39.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2026-02-22T20:19:39.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:39.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:39.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:39.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:39.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:39.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:39.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:39.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:39.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:39.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:39.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:39.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:39.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:39.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:39.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:39.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:39.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_26 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:39.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:39.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:39.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:39.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:39.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:39.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:39.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:39.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:39.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:39.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:39.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:39.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:39.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_26 2026-02-22T20:19:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '' 2026-02-22T20:19:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:39.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:39.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:39.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:39.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:39.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2026-02-22T20:19:39.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:39.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:39.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:39.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2026-02-22T20:19:39.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:39.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:39.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:39.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:39.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:39.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:39.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:39.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:39.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:39.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:39.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:39.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:39.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:39.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:39.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:39.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=11205' '!=' 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:39.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:39.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:19:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_26 2026-02-22T20:19:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_26 ']' 2026-02-22T20:19:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_26' 2026-02-22T20:19:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2026-02-22T20:19:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:40.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:40.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:40.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:40.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:40.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:40.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:40.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2026-02-22T20:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:40.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:40.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:40.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:40.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:40.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:40.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_26 2026-02-22T20:19:40.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:40.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:40.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:40.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:40.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:40.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:40.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:40.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:40.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:40.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:40.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_26 2026-02-22T20:19:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '' 2026-02-22T20:19:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:40.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:40.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:40.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:40.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:40.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:40.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:40.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:40.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2026-02-22T20:19:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2026-02-22T20:19:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:40.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:40.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:40.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:40.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:40.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:40.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:40.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:40.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:40.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:40.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:40.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=11205' '!=' 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:19:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_26 2026-02-22T20:19:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_26 ']' 2026-02-22T20:19:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_26' 2026-02-22T20:19:42.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2026-02-22T20:19:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2026-02-22T20:19:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:42.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:42.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:42.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:42.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2026-02-22T20:19:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:42.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:42.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:42.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:42.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:42.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_26 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:42.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:42.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:42.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:42.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:42.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:42.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_26 2026-02-22T20:19:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '' 2026-02-22T20:19:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:42.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:42.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:42.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:42.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:42.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:42.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:42.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:42.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:42.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:42.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2026-02-22T20:19:42.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:42.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:42.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:42.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2026-02-22T20:19:42.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:42.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:42.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:42.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:42.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:42.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:42.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:42.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:42.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:42.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:42.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:42.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:42.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:42.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:42.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:42.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:42.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:42.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=11205' '!=' 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:42.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:42.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_26 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_26 ']' 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_26' 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2026-02-22T20:19:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:44.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:44.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:44.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:44.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:44.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:44.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:44.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:44.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2026-02-22T20:19:44.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:44.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:44.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:44.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:44.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:44.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:44.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:44.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:44.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:44.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_26 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:45.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_26 2026-02-22T20:19:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:45.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '' 2026-02-22T20:19:45.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:45.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:45.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:45.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:45.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:45.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:45.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:45.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:45.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:45.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:45.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:45.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:45.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:45.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:45.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2026-02-22T20:19:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2026-02-22T20:19:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:45.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:45.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:45.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:45.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:45.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=11205], [object_number=8, tag_tid=1, entry_tid=11204], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:45.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:45.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:45.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:45.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=11205' '!=' 'object_number=9, tag_tid=1, entry_tid=11205' 2026-02-22T20:19:45.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:45.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:19:47.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_26 2026-02-22T20:19:47.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:47.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:47.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:47.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:47.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_26 ']' 2026-02-22T20:19:47.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_26' 2026-02-22T20:19:47.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2026-02-22T20:19:47.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:47.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:47.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:47.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:47.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:47.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:47.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:47.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:47.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:47.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:47.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:47.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:47.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:47.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:47.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:47.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:47.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:47.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:47.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:47.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:47.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:47.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-22T20:19:47.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:47.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_26 2026-02-22T20:19:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:47.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:47.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:47.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:47.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:47.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:47.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:47.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:47.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:47.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_26 2026-02-22T20:19:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:47.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_26 2026-02-22T20:19:47.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:47.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:47.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '' 2026-02-22T20:19:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:47.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:47.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:47.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:47.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:47.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:47.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:47.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:47.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2026-02-22T20:19:47.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:47.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2026-02-22T20:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_26.status 2026-02-22T20:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_26.status 2026-02-22T20:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:47.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:47.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2026-02-22T20:19:47.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:19:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:19:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:47.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11203], [object_number=10, tag_tid=1, entry_tid=11202]]], state=connected] 2026-02-22T20:19:47.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.status 2026-02-22T20:19:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:19:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:47.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2026-02-22T20:19:47.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:47.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:47.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:47.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:47.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:47.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:19:47.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:48.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2026-02-22T20:19:48.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_26': 2026-02-22T20:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11f83b800987 2026-02-22T20:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11f83b800987 2026-02-22T20:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:10:53 2026 2026-02-22T20:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:10:53 2026 2026-02-22T20:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:10:53 2026 2026-02-22T20:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11f83b800987 2026-02-22T20:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 48ef72e2-2819-48b5-b74b-f3e0f77e66c2 2026-02-22T20:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:19:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_26 snap 2026-02-22T20:19:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:19:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:19:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.export 2026-02-22T20:19:48.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_26.export 2026-02-22T20:19:48.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_26.export 2026-02-22T20:19:48.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.export 2026-02-22T20:19:48.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_26.export 2026-02-22T20:19:48.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:19:48.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_26.export 2026-02-22T20:19:48.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_26.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_26.export 2026-02-22T20:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_27 2026-02-22T20:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_27 snap 2026-02-22T20:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:19:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:19:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2026-02-22T20:19:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2026-02-22T20:19:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:51.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:19:51.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:19:51.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:51.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:19:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2026-02-22T20:19:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:19:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:19:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:19:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2026-02-22T20:19:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:19:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:19:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:19:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:19:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:19:52.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:52.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:19:52.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:52.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2026-02-22T20:19:52.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:19:52.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:52.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:52.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:52.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:52.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:52.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:52.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:52.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:19:52.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:19:52.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:52.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:52.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:52.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:52.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2026-02-22T20:19:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:19:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2026-02-22T20:19:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:19:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:19:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:19:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:52.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:52.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:52.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:52.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2026-02-22T20:19:53.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:53.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:19:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2026-02-22T20:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:53.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:53.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:53.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:53.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2026-02-22T20:19:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:19:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:19:53.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:19:53.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:19:53.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:19:53.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:19:53.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:19:53.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:53.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2026-02-22T20:19:53.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:53.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:53.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:53.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_27 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:53.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_27 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_27 ']' 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_27' 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:53.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:53.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:53.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2026-02-22T20:19:53.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:53.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:53.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:53.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:53.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:53.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:53.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2026-02-22T20:19:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:53.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:53.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:53.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:53.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:53.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:53.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:53.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:53.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:53.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:53.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:53.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:53.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:53.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_27 2026-02-22T20:19:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:53.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:53.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:53.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:53.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:53.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:53.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:53.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_27 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '' 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:53.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:53.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:53.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:53.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:53.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:53.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:53.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:53.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:53.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:53.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:53.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:53.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:53.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:53.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:53.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28101' '!=' '' 2026-02-22T20:19:53.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:19:53.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28101' = '' 2026-02-22T20:19:53.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:53.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:19:53.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:53.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:53.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:19:53.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:53.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:19:53.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:19:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:19:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:19:53.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:53.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:19:53.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:19:53.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28101 2026-02-22T20:19:53.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 28101 2026-02-22T20:19:53.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:19:53.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:53.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:19:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_27 2026-02-22T20:19:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:53.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_27 ']' 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_27' 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:53.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:53.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2026-02-22T20:19:53.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:53.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:53.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:53.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:53.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:53.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2026-02-22T20:19:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:53.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:53.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:53.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:53.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_27 2026-02-22T20:19:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:53.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:53.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:53.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:53.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:53.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:53.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_27 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '' 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:54.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:54.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:54.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:54.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:54.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:19:54.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2026-02-22T20:19:54.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:54.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:54.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:54.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2026-02-22T20:19:54.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:54.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:54.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:54.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:54.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:54.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:54.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:54.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:54.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:54.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:54.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:54.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:54.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:54.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:54.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:54.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28101' '!=' 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:54.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:54.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:19:54.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_27 2026-02-22T20:19:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_27 ']' 2026-02-22T20:19:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_27' 2026-02-22T20:19:54.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2026-02-22T20:19:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:54.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:54.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:54.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:54.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:54.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:54.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:54.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:54.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:54.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:54.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:54.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:54.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_27 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:54.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:54.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:54.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:54.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:54.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:54.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:54.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_27 2026-02-22T20:19:54.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '' 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:54.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:54.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:54.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:54.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:54.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:54.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:54.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:19:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2026-02-22T20:19:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2026-02-22T20:19:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:54.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:54.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:54.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:54.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:54.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:54.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:54.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:54.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28101' '!=' 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:54.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:54.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:19:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_27 2026-02-22T20:19:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_27 ']' 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_27' 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:55.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:55.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:55.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:55.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:55.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:55.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:55.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:55.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:55.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:55.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:55.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:55.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:55.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:55.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:55.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:55.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:55.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:55.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2026-02-22T20:19:55.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:55.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:55.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:55.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:55.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:55.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:55.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:55.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:55.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:55.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:55.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:55.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:55.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:55.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_27 2026-02-22T20:19:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:55.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:55.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:55.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:55.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:55.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:55.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:55.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:55.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_27 2026-02-22T20:19:55.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:55.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:55.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:55.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '' 2026-02-22T20:19:55.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:55.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:55.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:55.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:55.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:55.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:55.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:55.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:55.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:55.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:55.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:55.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:55.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:55.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:55.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:55.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:55.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:19:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2026-02-22T20:19:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2026-02-22T20:19:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:55.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:55.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:55.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:55.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:55.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:55.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:55.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:55.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:55.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28101' '!=' 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:55.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:55.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:19:57.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_27 2026-02-22T20:19:57.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:57.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:57.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_27 ']' 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_27' 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:57.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:57.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:57.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:57.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:57.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:57.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:57.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:57.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:57.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:57.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:57.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:57.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_27 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:19:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:19:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:57.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:19:57.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_27 2026-02-22T20:19:57.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '' 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:57.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:57.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:57.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:57.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:57.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:57.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:19:57.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:19:57.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:19:57.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:19:57.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:57.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:19:57.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:57.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:19:57.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:19:57.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:19:57.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:57.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28101], [object_number=24, tag_tid=1, entry_tid=28100], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:19:57.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:19:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28101' '!=' 'object_number=25, tag_tid=1, entry_tid=28101' 2026-02-22T20:19:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:19:57.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:19:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_27 2026-02-22T20:19:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:19:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_27 ']' 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_27' 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:19:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:19:59.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2026-02-22T20:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:19:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:19:59.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:19:59.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:59.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:59.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:19:59.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:59.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:59.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:59.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:19:59.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:19:59.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2026-02-22T20:19:59.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:19:59.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:19:59.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:19:59.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:19:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:19:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:19:59.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:19:59.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:59.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:59.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:19:59.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2026-02-22T20:19:59.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-22T20:19:59.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:19:59.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:00.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_27 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:20:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:00.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:00.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:00.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:00.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_27 2026-02-22T20:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_27 2026-02-22T20:20:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:20:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '' 2026-02-22T20:20:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:20:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:20:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:20:00.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:20:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:20:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:20:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:20:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:20:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:20:00.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:20:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:20:00.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:20:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2026-02-22T20:20:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:20:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2026-02-22T20:20:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:20:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:00.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_27.status 2026-02-22T20:20:00.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_27.status 2026-02-22T20:20:00.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:20:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2026-02-22T20:20:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:20:00.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:20:00.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:20:00.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:00.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:20:00.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28099], [object_number=26, tag_tid=1, entry_tid=28098]]], state=connected] 2026-02-22T20:20:00.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.status 2026-02-22T20:20:00.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:20:01.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2026-02-22T20:20:01.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_27': 2026-02-22T20:20:01.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:20:01.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:20:01.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:20:01.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11f87f0edf31 2026-02-22T20:20:01.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11f87f0edf31 2026-02-22T20:20:01.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:10:58 2026 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:10:58 2026 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:11:59 2026 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11f87f0edf31 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:20:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 06f8b9e5-d7ff-406a-bd82-58b730f2866b 2026-02-22T20:20:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:20:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_27 snap 2026-02-22T20:20:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:20:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:20:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.export 2026-02-22T20:20:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_27.export 2026-02-22T20:20:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_27.export 2026-02-22T20:20:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.export 2026-02-22T20:20:01.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:20:01.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_27.export 2026-02-22T20:20:01.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:20:01.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_27.export 2026-02-22T20:20:01.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_27.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_27.export 2026-02-22T20:20:01.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:01.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:20:01.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_28 2026-02-22T20:20:01.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_28 snap 2026-02-22T20:20:01.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:01.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:01.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:01.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:01.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:20:03.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:20:03.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:03.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:20:04.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2026-02-22T20:20:04.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:04.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:04.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:04.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:20:04.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:20:04.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:20:04.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2026-02-22T20:20:04.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:20:04.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:20:04.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:04.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:04.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:20:04.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:20:04.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:04.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:20:04.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:04.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:20:04.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:04.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2026-02-22T20:20:04.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:20:04.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:04.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:04.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:04.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:20:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:20:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:20:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:04.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2026-02-22T20:20:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2026-02-22T20:20:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:20:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:04.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:04.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:20:04.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:20:04.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:04.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:04.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:04.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:04.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2026-02-22T20:20:05.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:05.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:05.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:05.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:05.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2026-02-22T20:20:05.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:20:05.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:05.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:05.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:05.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:05.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:20:05.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:20:05.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:05.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:05.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:05.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:05.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2026-02-22T20:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:05.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2026-02-22T20:20:05.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:20:05.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:05.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:05.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:05.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:05.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:05.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:05.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:05.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:05.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2026-02-22T20:20:05.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:20:05.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:20:05.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:20:05.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:20:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:20:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:20:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:20:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2026-02-22T20:20:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:20:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:20:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_28 2026-02-22T20:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:20:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_28 2026-02-22T20:20:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_28 ']' 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_28' 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:05.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:05.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2026-02-22T20:20:05.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:05.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:05.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:05.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:05.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:05.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:05.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:05.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:05.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:05.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:05.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:05.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:05.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:05.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:05.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:05.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:05.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:05.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:05.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:05.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:05.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:05.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:05.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:05.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:05.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:05.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_28 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:05.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:05.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:05.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:05.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:05.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:05.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_28 2026-02-22T20:20:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:20:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:05.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:05.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:05.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:05.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:05.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:05.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:05.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_28 2026-02-22T20:20:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_28 '' 2026-02-22T20:20:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_28.status 2026-02-22T20:20:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_28.status 2026-02-22T20:20:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2026-02-22T20:20:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:05.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:20:05.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:20:05.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:05.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:05.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034], [object_number=17, tag_tid=1, entry_tid=18033], [object_number=16, tag_tid=1, entry_tid=18032]]], state=connected] 2026-02-22T20:20:05.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:05.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:20:05.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2026-02-22T20:20:05.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:05.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:05.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:05.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2026-02-22T20:20:05.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:05.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:05.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:05.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:05.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_28.status 2026-02-22T20:20:05.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_28.status 2026-02-22T20:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2026-02-22T20:20:05.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:05.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:20:05.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:20:05.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:05.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:05.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034], [object_number=17, tag_tid=1, entry_tid=18033], [object_number=16, tag_tid=1, entry_tid=18032]]], state=connected] 2026-02-22T20:20:05.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:05.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:05.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:05.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=18035' '!=' '' 2026-02-22T20:20:05.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:20:05.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=18035' = '' 2026-02-22T20:20:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:20:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:20:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:20:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:20:05.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:05.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:20:05.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:20:05.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:20:05.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:20:05.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:20:05.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:20:05.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:20:05.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:05.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:20:05.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:20:05.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=18035 2026-02-22T20:20:05.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 18035 2026-02-22T20:20:05.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:20:05.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:05.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:20:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_28 2026-02-22T20:20:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_28 ']' 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_28' 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2026-02-22T20:20:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:05.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:05.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:05.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:05.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:05.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2026-02-22T20:20:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:06.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:06.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:06.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:06.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:06.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_28 2026-02-22T20:20:06.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:20:06.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:06.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:06.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:20:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:06.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:06.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:06.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:06.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_28 2026-02-22T20:20:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:06.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:06.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:06.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:06.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:06.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:06.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:06.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:06.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:06.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:06.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_28 2026-02-22T20:20:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_28 '' 2026-02-22T20:20:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:06.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_28.status 2026-02-22T20:20:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_28.status 2026-02-22T20:20:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2026-02-22T20:20:06.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:06.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:20:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:20:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034], [object_number=17, tag_tid=1, entry_tid=18033], [object_number=16, tag_tid=1, entry_tid=18032]]], state=connected] 2026-02-22T20:20:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:20:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2026-02-22T20:20:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2026-02-22T20:20:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_28.status 2026-02-22T20:20:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_28.status 2026-02-22T20:20:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2026-02-22T20:20:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:20:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:20:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034], [object_number=17, tag_tid=1, entry_tid=18033], [object_number=16, tag_tid=1, entry_tid=18032]]], state=connected] 2026-02-22T20:20:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=18035' '!=' 'object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:20:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_28 2026-02-22T20:20:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_28 ']' 2026-02-22T20:20:06.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_28' 2026-02-22T20:20:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2026-02-22T20:20:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2026-02-22T20:20:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:06.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:06.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:06.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:06.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:06.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:06.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:06.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:06.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:06.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2026-02-22T20:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:06.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:06.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:06.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:06.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_28 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:20:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:06.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:06.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:06.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:06.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:06.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:06.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:06.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_28 2026-02-22T20:20:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:20:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:20:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:06.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:06.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:06.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:06.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:07.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:07.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_28 2026-02-22T20:20:07.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:07.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_28 '' 2026-02-22T20:20:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_28.status 2026-02-22T20:20:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_28.status 2026-02-22T20:20:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2026-02-22T20:20:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:07.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:20:07.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:20:07.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:07.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:07.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034], [object_number=17, tag_tid=1, entry_tid=18033], [object_number=16, tag_tid=1, entry_tid=18032]]], state=connected] 2026-02-22T20:20:07.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:07.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:20:07.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2026-02-22T20:20:07.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:07.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:07.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:07.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2026-02-22T20:20:07.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:07.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:07.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:07.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:07.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_28.status 2026-02-22T20:20:07.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_28.status 2026-02-22T20:20:07.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:07.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:07.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2026-02-22T20:20:07.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:07.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:20:07.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:20:07.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034], [object_number=17, tag_tid=1, entry_tid=18033], [object_number=16, tag_tid=1, entry_tid=18032]]], state=connected] 2026-02-22T20:20:07.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:07.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:07.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:07.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=18035' '!=' 'object_number=19, tag_tid=1, entry_tid=18035' 2026-02-22T20:20:07.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:07.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:20:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_28 2026-02-22T20:20:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_28 ']' 2026-02-22T20:20:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_28' 2026-02-22T20:20:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2026-02-22T20:20:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:07.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:07.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:07.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:07.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:07.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2026-02-22T20:20:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:07.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2026-02-22T20:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:07.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:07.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:07.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:07.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:08.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:08.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_28 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:08.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:08.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:08.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:08.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:08.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-22T20:20:08.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:08.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_28 2026-02-22T20:20:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:20:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:08.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:08.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:08.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:08.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_28 2026-02-22T20:20:08.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:08.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_28 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_28 '' 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_28.status 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_28.status 2026-02-22T20:20:08.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:08.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:08.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2026-02-22T20:20:08.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:08.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:20:08.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:20:08.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:08.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:08.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:08.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:08.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:20:08.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2026-02-22T20:20:08.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:08.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:08.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:08.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2026-02-22T20:20:08.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:08.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:08.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:20:08.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:08.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_28.status 2026-02-22T20:20:08.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_28.status 2026-02-22T20:20:08.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:08.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:08.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2026-02-22T20:20:08.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:08.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:20:08.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:20:08.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:08.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:08.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18035], [object_number=18, tag_tid=1, entry_tid=18034]]], state=connected] 2026-02-22T20:20:08.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.status 2026-02-22T20:20:08.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:20:08.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:20:08.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:08.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2026-02-22T20:20:08.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:08.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:08.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:08.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:08.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:20:08.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:20:08.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:20:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2026-02-22T20:20:09.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_28': 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11f829a1601d 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11f829a1601d 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:11:07 2026 2026-02-22T20:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:11:07 2026 2026-02-22T20:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:11:07 2026 2026-02-22T20:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11f829a1601d 2026-02-22T20:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 87c1917e-ada7-442a-8193-765ec0601b5e 2026-02-22T20:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:20:09.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_28 snap 2026-02-22T20:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.export 2026-02-22T20:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_28.export 2026-02-22T20:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_28.export 2026-02-22T20:20:09.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.export 2026-02-22T20:20:09.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:20:09.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_28.export 2026-02-22T20:20:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:20:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_28.export 2026-02-22T20:20:09.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_28.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_28.export 2026-02-22T20:20:09.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:09.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:20:09.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_29 2026-02-22T20:20:09.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_29 snap 2026-02-22T20:20:09.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:09.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:09.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:09.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:09.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:20:11.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2026-02-22T20:20:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:20:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:20:12.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:20:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:20:12.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2026-02-22T20:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:20:12.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:12.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:20:12.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:12.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2026-02-22T20:20:12.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:12.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:12.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:12.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:20:12.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:20:12.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:12.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:12.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:12.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:12.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2026-02-22T20:20:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2026-02-22T20:20:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2026-02-22T20:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:20:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:13.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:13.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:13.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:13.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2026-02-22T20:20:13.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:20:13.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:20:13.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:20:13.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:20:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_29 2026-02-22T20:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:20:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_29 2026-02-22T20:20:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_29 ']' 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_29' 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:13.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:13.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:13.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2026-02-22T20:20:13.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:13.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2026-02-22T20:20:13.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:13.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2026-02-22T20:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:13.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:13.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:13.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:13.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_29 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '' 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:13.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:13.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:13.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:13.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:13.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:13.570 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2579610357451241) > weight (0.25) 2026-02-22T20:20:13.571 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7068423268780746) > weight (0.25) 2026-02-22T20:20:13.571 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2026-02-22T20:20:13.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2026-02-22T20:20:13.571 INFO:tasks.rbd_mirror_thrash:waiting for 9.951587017591958 secs before reviving daemons 2026-02-22T20:20:13.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:20:13.560+0000 7f1efb464640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 84382) UID: 1000 2026-02-22T20:20:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:13.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:13.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:13.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2026-02-22T20:20:13.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:13.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:13.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:13.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2026-02-22T20:20:13.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:13.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:13.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:13.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:13.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:13.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:13.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:13.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:13.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:13.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=39546' '!=' '' 2026-02-22T20:20:13.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:20:13.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=39546' = '' 2026-02-22T20:20:13.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:13.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:13.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:20:13.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:20:13.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:20:13.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:13.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:20:13.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:20:13.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:20:13.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:13.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:20:13.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:20:13.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:20:13.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:13.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:20:13.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:20:13.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39546 2026-02-22T20:20:13.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 39546 2026-02-22T20:20:13.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:20:13.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:13.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:20:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_29 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_29 ']' 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_29' 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:13.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:13.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:13.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:13.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:13.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:13.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:13.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:13.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:13.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:14.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2026-02-22T20:20:14.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:14.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:14.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:14.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2026-02-22T20:20:14.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:14.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:14.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:14.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:14.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:14.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:14.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_29 2026-02-22T20:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '' 2026-02-22T20:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:14.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:14.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:14.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:14.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:14.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:14.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:14.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:14.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2026-02-22T20:20:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2026-02-22T20:20:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:14.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:14.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:14.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:14.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=39546' '!=' 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:20:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_29 2026-02-22T20:20:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_29 ']' 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_29' 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2026-02-22T20:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:14.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:14.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:14.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:14.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:14.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2026-02-22T20:20:14.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:14.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:14.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:14.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2026-02-22T20:20:14.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:14.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:14.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:14.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:14.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:14.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:14.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:14.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:14.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:14.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:14.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:14.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_29 2026-02-22T20:20:14.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:14.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:14.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:14.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '' 2026-02-22T20:20:14.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:14.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:14.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:14.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:14.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:14.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:14.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:14.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2026-02-22T20:20:14.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:14.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:14.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:14.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2026-02-22T20:20:14.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:14.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:14.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:14.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:14.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:14.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:14.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:14.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:14.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:14.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:14.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:14.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:14.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=39546' '!=' 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:14.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:14.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_29 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_29 ']' 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_29' 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:15.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:15.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:15.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:15.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:15.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:15.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:15.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:15.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:15.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:15.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:15.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:15.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:15.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:15.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:15.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:15.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:15.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:15.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:15.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:15.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2026-02-22T20:20:15.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:15.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:15.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_29 2026-02-22T20:20:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '' 2026-02-22T20:20:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:15.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:15.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:15.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:15.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:15.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:15.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:15.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:15.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:15.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:15.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:15.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:15.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:15.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:15.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2026-02-22T20:20:15.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:15.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:15.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:15.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2026-02-22T20:20:15.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:15.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:15.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:15.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:15.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:15.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:15.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:15.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:15.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:15.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:15.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:15.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:15.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:15.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:15.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:15.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:15.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:15.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:15.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=39546' '!=' 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:15.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:15.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:20:17.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_29 2026-02-22T20:20:17.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_29 ']' 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_29' 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:17.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:17.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:17.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:17.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:17.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:17.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:17.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:17.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:17.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2026-02-22T20:20:17.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:17.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:17.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:17.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:17.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:17.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:17.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:17.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:17.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:17.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:17.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:17.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:17.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:17.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:17.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:17.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_29 2026-02-22T20:20:17.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:17.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:17.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:17.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '' 2026-02-22T20:20:17.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:17.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:17.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:17.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:17.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:17.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:17.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:17.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:17.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:17.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:17.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:17.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:17.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2026-02-22T20:20:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2026-02-22T20:20:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:17.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:17.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:17.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:17.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:17.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=39546], [object_number=37, tag_tid=1, entry_tid=39545], [object_number=36, tag_tid=1, entry_tid=39544], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:17.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:17.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:17.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:17.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=39546' '!=' 'object_number=38, tag_tid=1, entry_tid=39546' 2026-02-22T20:20:17.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:17.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:20:19.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_29 2026-02-22T20:20:19.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:19.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:19.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:19.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:19.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_29 ']' 2026-02-22T20:20:19.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_29' 2026-02-22T20:20:19.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2026-02-22T20:20:19.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:19.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:19.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:19.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:19.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:19.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:19.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:19.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:19.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:19.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:19.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2026-02-22T20:20:19.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:19.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:19.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:19.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:19.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:19.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:19.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:19.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2026-02-22T20:20:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:19.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:19.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:19.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:19.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2026-02-22T20:20:19.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:19.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_29 2026-02-22T20:20:19.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '' 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:19.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:19.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:19.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:19.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:19.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:19.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:19.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:19.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:19.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:19.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:19.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_29.status 2026-02-22T20:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_29.status 2026-02-22T20:20:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2026-02-22T20:20:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:19.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:19.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:20:19.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:19.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:19.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=39546], [object_number=39, tag_tid=1, entry_tid=39543]]], state=connected] 2026-02-22T20:20:19.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.status 2026-02-22T20:20:19.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:19.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:20:19.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:19.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2026-02-22T20:20:19.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:19.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:19.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:19.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:19.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:20:19.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:20:19.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:20:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2026-02-22T20:20:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_29': 2026-02-22T20:20:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:20:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:20:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:20:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aa3f75f918 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aa3f75f918 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:11:14 2026 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:11:14 2026 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:12:15 2026 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aa3f75f918 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:20:20.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:20:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 3347a300-05b4-4e41-a348-8ac79a55ae81 2026-02-22T20:20:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:20:20.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:20.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_29 snap 2026-02-22T20:20:20.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:20.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:20:20.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:20.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:20:20.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.export 2026-02-22T20:20:20.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_29.export 2026-02-22T20:20:20.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_29.export 2026-02-22T20:20:20.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.export 2026-02-22T20:20:21.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:20:21.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_29.export 2026-02-22T20:20:21.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:20:21.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_29.export 2026-02-22T20:20:21.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_29.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_29.export 2026-02-22T20:20:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:20:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_30 2026-02-22T20:20:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_30 snap 2026-02-22T20:20:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:20:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:21.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:20:23.523 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2026-02-22T20:20:23.524 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:20:23.524 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2026-02-22T20:20:23.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2026-02-22T20:20:23.524 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2026-02-22T20:20:23.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2026-02-22T20:20:23.527 INFO:tasks.rbd_mirror_thrash:waiting for 103.70198196906325 secs before thrashing 2026-02-22T20:20:23.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2026-02-22T20:20:23.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:23.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:23.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:20:23.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2026-02-22T20:20:23.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:23.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:23.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:20:23.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:20:23.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:20:23.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:23.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:20:24.311 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5317900017717704) > weight (0.25) 2026-02-22T20:20:24.312 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-22T20:20:24.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-22T20:20:24.312 INFO:tasks.rbd_mirror_thrash:waiting for 5.566334408351614 secs before reviving daemons 2026-02-22T20:20:24.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:20:24.300+0000 7f8988de5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 94940) UID: 1000 2026-02-22T20:20:24.352 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:20:24.340+0000 7f557ffc8640 -1 reset not still connected to 0x55921182e340 2026-02-22T20:20:24.352 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:20:24.340+0000 7f557ffc8640 -1 reset not still connected to 0x55921fb752b0 2026-02-22T20:20:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2026-02-22T20:20:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:20:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:20:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:20:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:20:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2026-02-22T20:20:24.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:20:24.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:20:24.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:24.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:24.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:20:24.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:20:24.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:24.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:20:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:24.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:20:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2026-02-22T20:20:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:20:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:20:24.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:24.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:24.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:24.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2026-02-22T20:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:25.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:25.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:25.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:25.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:25.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:25.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:20:25.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:20:25.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:25.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2026-02-22T20:20:25.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2026-02-22T20:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:25.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:25.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:25.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:25.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2026-02-22T20:20:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2026-02-22T20:20:25.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:25.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:25.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 1 2026-02-22T20:20:26.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:26.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:20:26.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:26.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:20:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:20:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:26.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:26.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:26.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:26.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2026-02-22T20:20:26.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:26.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:20:26.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:20:26.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:26.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:26.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:26.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:26.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2026-02-22T20:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:20:26.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:26.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:20:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:26.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:26.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:26.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:26.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2026-02-22T20:20:26.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:26.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:26.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:26.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:26.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2026-02-22T20:20:26.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:20:26.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:26.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:26.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:26.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:26.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:26.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:26.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:20:26.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:20:26.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:26.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:26.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:26.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:26.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2026-02-22T20:20:26.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:26.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:26.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:26.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:26.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 2 2026-02-22T20:20:28.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:28.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:20:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2026-02-22T20:20:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:20:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:28.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:20:28.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:20:28.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:28.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:28.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:28.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:28.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2026-02-22T20:20:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:28.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:28.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:20:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:20:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:20:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:28.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:28.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:28.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:28.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2026-02-22T20:20:28.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2026-02-22T20:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:28.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:28.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:28.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:28.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:20:28.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:20:28.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:28.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:28.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:28.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:28.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2026-02-22T20:20:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:28.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:28.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:20:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:20:28.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:28.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2026-02-22T20:20:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:28.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:28.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:28.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 4 2026-02-22T20:20:29.880 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-22T20:20:29.880 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:20:29.881 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-22T20:20:29.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-22T20:20:29.881 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-22T20:20:29.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-22T20:20:29.884 INFO:tasks.rbd_mirror_thrash:waiting for 86.6559545109354 secs before thrashing 2026-02-22T20:20:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2026-02-22T20:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:20:32.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:20:32.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:32.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:32.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:32.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:32.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2026-02-22T20:20:32.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:32.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:32.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:32.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:32.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:32.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:32.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:32.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:32.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:20:32.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:20:32.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:32.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:32.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:32.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:32.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2026-02-22T20:20:32.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:32.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:32.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:32.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:32.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2026-02-22T20:20:32.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:20:32.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:32.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:32.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:20:32.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:20:32.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:32.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:32.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:32.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:32.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2026-02-22T20:20:32.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:32.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:32.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:32.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:32.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2026-02-22T20:20:32.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:20:32.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:32.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:32.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:32.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:32.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:32.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:32.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:32.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:32.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:32.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:32.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:32.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:32.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:32.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:32.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:20:32.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:32.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:32.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:32.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:20:32.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:32.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:32.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:32.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:32.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:32.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:32.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:32.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:20:32.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:20:32.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:32.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:32.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:32.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:32.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2026-02-22T20:20:33.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:33.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:33.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:33.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:33.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 8 2026-02-22T20:20:41.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:41.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:20:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:20:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:41.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:41.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:41.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:41.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2026-02-22T20:20:41.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:41.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:41.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:41.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:41.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2026-02-22T20:20:41.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:20:41.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:41.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:41.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:41.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:41.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:41.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:41.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:20:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:41.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:41.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:41.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:41.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2026-02-22T20:20:41.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:41.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:41.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:41.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:41.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2026-02-22T20:20:41.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:20:41.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:41.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:41.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:20:41.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:20:41.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2026-02-22T20:20:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2026-02-22T20:20:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:20:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:20:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:20:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:41.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:41.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:41.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:41.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2026-02-22T20:20:41.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:41.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:41.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 8 2026-02-22T20:20:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:20:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:20:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:49.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:49.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:49.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:49.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2026-02-22T20:20:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2026-02-22T20:20:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:20:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:20:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:20:49.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:49.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:49.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:49.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:49.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2026-02-22T20:20:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:20:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:20:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:20:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:20:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:20:49.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:49.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:49.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:20:49.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2026-02-22T20:20:49.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:49.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:49.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:49.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:49.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2026-02-22T20:20:49.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:20:49.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:49.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:49.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:20:49.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:20:49.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:49.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:49.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:20:49.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:20:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:20:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2026-02-22T20:20:49.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:49.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:49.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:49.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:49.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 8 2026-02-22T20:20:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:20:57.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:57.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:57.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:57.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:57.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:57.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:57.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:20:57.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:20:57.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:57.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:57.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:57.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2026-02-22T20:20:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:20:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:57.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:57.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2026-02-22T20:20:57.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:20:57.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:20:57.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:57.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:57.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:57.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:20:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:20:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:57.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:57.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:57.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:57.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2026-02-22T20:20:57.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:20:57.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:20:57.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:20:57.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:20:57.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2026-02-22T20:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_30 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:20:57.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:20:57.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_30 2026-02-22T20:20:57.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:57.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:57.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_30 ']' 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_30' 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:20:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:20:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:20:57.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:20:57.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:57.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:57.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:20:57.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2026-02-22T20:20:58.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-22T20:20:58.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:20:58.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:20:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:20:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:20:58.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:58.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:58.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:20:58.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2026-02-22T20:20:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:58.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_30 2026-02-22T20:20:58.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:58.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:58.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_30 2026-02-22T20:20:58.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_30 '' 2026-02-22T20:20:58.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:58.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:58.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_30 2026-02-22T20:20:58.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:20:58.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_30.status 2026-02-22T20:20:58.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_30.status 2026-02-22T20:20:58.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:58.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.status 2026-02-22T20:20:58.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2026-02-22T20:20:58.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.status 2026-02-22T20:20:58.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:58.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:20:58.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:58.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=41775], [object_number=38, tag_tid=1, entry_tid=41774]]], state=connected] 2026-02-22T20:20:58.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=41775], [object_number=38, tag_tid=1, entry_tid=41774]]], state=connected] 2026-02-22T20:20:58.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.status 2026-02-22T20:20:58.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_30 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_30 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_30.status 2026-02-22T20:20:58.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_30.status 2026-02-22T20:20:58.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:20:58.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.status 2026-02-22T20:20:58.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2026-02-22T20:20:58.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.status 2026-02-22T20:20:58.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:20:58.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 10 2026-02-22T20:20:58.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:20:58.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=41775], [object_number=38, tag_tid=1, entry_tid=41774]]], state=connected] 2026-02-22T20:20:58.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=41, tag_tid=2, entry_tid=1], [object_number=40, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=41775], [object_number=38, tag_tid=1, entry_tid=41774]]], state=connected] 2026-02-22T20:20:58.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.status 2026-02-22T20:20:58.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=41, tag_tid=2, entry_tid=1' -a 'object_number=41, tag_tid=2, entry_tid=1' = 'object_number=41, tag_tid=2, entry_tid=1' 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:20:58.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:20:59.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2026-02-22T20:20:59.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_30': 2026-02-22T20:20:59.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:20:59.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:20:59.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:20:59.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 112721037344 2026-02-22T20:20:59.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.112721037344 2026-02-22T20:20:59.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:20:59.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:20:59.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:20:59.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:20:59.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:11:26 2026 2026-02-22T20:20:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:11:26 2026 2026-02-22T20:20:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:12:27 2026 2026-02-22T20:20:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:20:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 112721037344 2026-02-22T20:20:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:20:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:20:59.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 54a77260-ad89-4142-8ad2-272fddc13d51 2026-02-22T20:20:59.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:20:59.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:59.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_30 snap 2026-02-22T20:20:59.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:59.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:20:59.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:59.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:20:59.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.export 2026-02-22T20:20:59.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_30.export 2026-02-22T20:20:59.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_30.export 2026-02-22T20:20:59.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.export 2026-02-22T20:20:59.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:20:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_30.export 2026-02-22T20:20:59.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:20:59.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_30.export 2026-02-22T20:20:59.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_30.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_30.export 2026-02-22T20:20:59.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:20:59.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:20:59.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_31 2026-02-22T20:20:59.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_31 snap 2026-02-22T20:20:59.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:20:59.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:20:59.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:20:59.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:20:59.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2026-02-22T20:21:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:21:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:21:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:21:02.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2026-02-22T20:21:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:21:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:21:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:21:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:21:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:21:02.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:02.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:21:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:21:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:02.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:02.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:02.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:02.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2026-02-22T20:21:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:02.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:21:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:21:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:02.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2026-02-22T20:21:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2026-02-22T20:21:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:21:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:21:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:21:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:21:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:02.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:02.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:02.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:02.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2026-02-22T20:21:03.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:03.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:03.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:03.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_31 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:03.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:03.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_31 2026-02-22T20:21:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:21:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:21:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:21:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:21:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:21:03.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:21:03.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_31 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:21:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:21:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_31 2026-02-22T20:21:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_31 ']' 2026-02-22T20:21:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_31' 2026-02-22T20:21:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2026-02-22T20:21:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2026-02-22T20:21:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:03.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:03.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:03.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:03.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:03.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:03.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:03.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:03.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:03.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:03.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:03.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:03.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:03.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:03.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:03.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_31 2026-02-22T20:21:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:03.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:03.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:03.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:03.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:03.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_31 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:03.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:03.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:03.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:03.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:03.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:03.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:03.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:03.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:03.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:03.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:03.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:03.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:03.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:03.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_31 2026-02-22T20:21:03.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:03.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:03.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:03.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '' 2026-02-22T20:21:03.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:03.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:03.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:03.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:03.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:03.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:03.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:03.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:03.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:03.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:03.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:03.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:03.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:03.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:03.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:03.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:03.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:03.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:03.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:03.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:03.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:03.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:03.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:03.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:03.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=30858' '!=' '' 2026-02-22T20:21:03.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:21:03.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=30858' = '' 2026-02-22T20:21:03.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:03.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:03.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:03.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:03.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:21:03.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:03.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:03.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:21:03.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:03.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:03.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:03.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:21:03.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:03.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:03.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:03.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30858 2026-02-22T20:21:03.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 30858 2026-02-22T20:21:03.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:03.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:03.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_31 2026-02-22T20:21:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_31 ']' 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_31' 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:03.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:03.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:03.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:03.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:03.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:03.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:03.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:03.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:03.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:03.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:03.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:03.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:03.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:03.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:04.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:04.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:04.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_31 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:04.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:04.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:04.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:04.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:04.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_31 2026-02-22T20:21:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:04.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:04.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:04.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:04.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_31 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '' 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:04.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:04.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:04.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:04.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:04.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:04.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:04.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2026-02-22T20:21:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2026-02-22T20:21:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:04.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:04.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:04.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:04.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:04.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:04.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:04.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:04.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:04.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=30858' '!=' 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:21:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_31 2026-02-22T20:21:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_31 ']' 2026-02-22T20:21:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_31' 2026-02-22T20:21:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2026-02-22T20:21:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2026-02-22T20:21:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:04.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:04.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:04.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:04.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:04.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:04.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:04.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:04.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:04.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:04.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:04.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:04.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:04.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:04.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:04.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:04.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:04.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:04.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:04.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:04.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:04.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:04.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:04.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:04.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:04.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:04.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:04.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:04.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:04.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:04.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:04.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:04.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:04.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:04.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:04.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:04.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_31 2026-02-22T20:21:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:04.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:04.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:04.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:04.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:04.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:04.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:04.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:04.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:04.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:04.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:04.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_31 2026-02-22T20:21:04.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:04.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:04.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:04.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:04.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:04.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:04.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:04.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:04.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:04.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_31 2026-02-22T20:21:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '' 2026-02-22T20:21:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:04.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:04.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:04.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:04.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:04.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:04.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:04.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:04.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:04.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:04.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2026-02-22T20:21:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:04.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:04.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:04.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2026-02-22T20:21:04.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:04.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:04.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:04.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:04.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:04.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:04.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:05.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:05.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:05.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:05.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:05.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:05.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:05.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:05.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:05.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=30858' '!=' 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:05.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:05.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:21:05.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_31 2026-02-22T20:21:05.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_31 ']' 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_31' 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:05.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:05.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:05.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:05.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:05.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:05.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2026-02-22T20:21:05.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:05.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:05.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:05.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:05.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:05.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:05.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:05.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:05.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2026-02-22T20:21:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:05.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_31 2026-02-22T20:21:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:05.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:05.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:05.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:05.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:06.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_31 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:06.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:06.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:06.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:06.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_31 2026-02-22T20:21:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '' 2026-02-22T20:21:06.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:06.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:06.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:06.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:06.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:06.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:06.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:06.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:06.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:06.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:06.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2026-02-22T20:21:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2026-02-22T20:21:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:06.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:06.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:06.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:06.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:06.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:06.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:06.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:06.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:06.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:06.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:06.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:06.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:06.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:06.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:06.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:06.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:06.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=30858' '!=' 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:06.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:06.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:21:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_31 2026-02-22T20:21:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_31 ']' 2026-02-22T20:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_31' 2026-02-22T20:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2026-02-22T20:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:07.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:07.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:07.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:07.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2026-02-22T20:21:07.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:07.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:07.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:07.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:07.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:07.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:07.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:07.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:07.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:07.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:07.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:07.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:07.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:07.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:07.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:07.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:07.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:07.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:07.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:07.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:07.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:07.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:07.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:07.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:07.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:07.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:07.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:07.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:07.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:07.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:07.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:07.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:07.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:07.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:07.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:07.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:07.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:07.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:07.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2026-02-22T20:21:07.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:07.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:07.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:07.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:07.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:07.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:07.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:07.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:07.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:07.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:07.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:07.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_31 2026-02-22T20:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:07.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_31 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:08.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:08.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:08.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:08.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:08.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:08.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:08.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_31 2026-02-22T20:21:08.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:08.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:08.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:08.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '' 2026-02-22T20:21:08.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:08.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:08.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:08.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:08.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:08.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:08.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:08.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:08.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:08.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:08.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:08.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:08.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:08.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:08.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:08.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:08.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:08.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2026-02-22T20:21:08.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:08.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:08.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:08.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=30, tag_tid=1, entry_tid=30858], [object_number=29, tag_tid=1, entry_tid=30857], [object_number=28, tag_tid=1, entry_tid=30856], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:08.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:08.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:08.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:08.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=30, tag_tid=1, entry_tid=30858' '!=' 'object_number=30, tag_tid=1, entry_tid=30858' 2026-02-22T20:21:08.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:08.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:21:10.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_31 2026-02-22T20:21:10.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_31 ']' 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_31' 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:10.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:10.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:10.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:10.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:10.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:10.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:10.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:10.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:10.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:10.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:10.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:10.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2026-02-22T20:21:10.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:10.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:10.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:10.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:10.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:10.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:10.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:10.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:10.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:10.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:10.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_31 2026-02-22T20:21:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:10.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:10.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-22T20:21:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:10.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_31 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:10.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:10.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:10.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:10.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_31 2026-02-22T20:21:10.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_31 2026-02-22T20:21:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '' 2026-02-22T20:21:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:10.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:10.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:10.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:10.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:10.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:10.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:10.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:10.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:10.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2026-02-22T20:21:10.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:10.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:10.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:10.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2026-02-22T20:21:10.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:10.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:10.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:21:10.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:10.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_31.status 2026-02-22T20:21:10.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_31.status 2026-02-22T20:21:10.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:10.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2026-02-22T20:21:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:10.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:10.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:10.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:10.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:10.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=30, tag_tid=1, entry_tid=30858], [object_number=31, tag_tid=1, entry_tid=30855]]], state=connected] 2026-02-22T20:21:10.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.status 2026-02-22T20:21:10.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:21:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:11.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2026-02-22T20:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_31': 2026-02-22T20:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e32242d4c 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e32242d4c 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:11:39 2026 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:11:39 2026 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:12:40 2026 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e32242d4c 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 581fb2af-3dc5-4555-9bcc-21afc60cd316 2026-02-22T20:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:21:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_31 snap 2026-02-22T20:21:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:21:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:21:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.export 2026-02-22T20:21:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_31.export 2026-02-22T20:21:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_31.export 2026-02-22T20:21:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.export 2026-02-22T20:21:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:11.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_31.export 2026-02-22T20:21:11.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:11.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_31.export 2026-02-22T20:21:11.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_31.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_31.export 2026-02-22T20:21:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:21:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_32 2026-02-22T20:21:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_32 snap 2026-02-22T20:21:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:12.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2026-02-22T20:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2026-02-22T20:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:14.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:14.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:21:14.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:14.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:14.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2026-02-22T20:21:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:15.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:15.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:21:15.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:21:15.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:21:15.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2026-02-22T20:21:15.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:21:15.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:21:15.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:15.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:15.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:21:15.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:21:15.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:15.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:21:15.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:15.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:21:15.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:15.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2026-02-22T20:21:15.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:15.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:15.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:15.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:21:15.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:21:15.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:15.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:15.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:15.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2026-02-22T20:21:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:21:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:21:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:21:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:21:15.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:21:15.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_32 2026-02-22T20:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:16.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_32 2026-02-22T20:21:16.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:16.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:16.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:16.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:16.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_32 ']' 2026-02-22T20:21:16.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_32' 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:16.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:16.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:16.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2026-02-22T20:21:16.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:16.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:16.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:16.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:16.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:16.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:16.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:16.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:16.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:16.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:16.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:16.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2026-02-22T20:21:16.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:16.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_32 2026-02-22T20:21:16.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:16.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '' 2026-02-22T20:21:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:16.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:16.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:16.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:16.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:16.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:16.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:16.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:16.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:16.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2026-02-22T20:21:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2026-02-22T20:21:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:16.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:16.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:16.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:16.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:16.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=14594' '!=' '' 2026-02-22T20:21:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:21:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=14594' = '' 2026-02-22T20:21:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:16.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:16.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:16.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:16.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:21:16.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:16.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:16.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:16.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:21:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:21:16.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:16.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:16.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:16.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=14594 2026-02-22T20:21:16.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 14594 2026-02-22T20:21:16.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:16.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:16.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:16.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_32 2026-02-22T20:21:16.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:16.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:16.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_32 ']' 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_32' 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:16.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:16.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:16.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:16.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:16.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:16.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:16.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2026-02-22T20:21:16.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:16.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_32 2026-02-22T20:21:16.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:16.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '' 2026-02-22T20:21:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:16.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:16.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:16.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:16.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:16.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:16.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:16.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2026-02-22T20:21:16.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:16.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:16.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:16.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2026-02-22T20:21:16.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:16.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:16.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:16.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:16.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:16.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:16.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:16.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:16.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:16.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:16.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:16.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:16.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:16.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:16.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:16.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=14594' '!=' 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:16.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:16.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:21:17.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_32 2026-02-22T20:21:17.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_32 ']' 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_32' 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:17.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:17.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:17.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:17.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2026-02-22T20:21:17.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:17.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:17.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:17.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:17.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:17.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:17.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:17.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:17.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:17.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:17.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:17.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:17.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2026-02-22T20:21:17.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:17.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_32 2026-02-22T20:21:17.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:17.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:17.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:17.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '' 2026-02-22T20:21:17.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:17.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:17.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:17.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:17.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:17.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:17.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:17.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:17.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:17.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:17.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:17.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:17.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:17.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:17.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:17.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:17.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:17.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:17.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:17.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:17.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:17.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:17.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:17.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:17.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:17.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:17.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:17.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=14594' '!=' 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:17.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:17.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:21:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_32 2026-02-22T20:21:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_32 ']' 2026-02-22T20:21:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_32' 2026-02-22T20:21:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2026-02-22T20:21:17.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:17.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:17.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:17.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:17.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2026-02-22T20:21:18.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_32 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '' 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:18.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:18.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:18.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:18.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:18.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:18.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:18.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:18.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:18.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2026-02-22T20:21:18.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:18.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:18.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:18.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2026-02-22T20:21:18.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:18.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:18.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:18.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:18.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:18.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:18.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:18.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:18.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:18.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:18.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:18.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:18.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:18.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:18.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:18.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:18.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:18.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:18.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=14594' '!=' 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:18.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:18.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:21:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_32 2026-02-22T20:21:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:19.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_32 ']' 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_32' 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:19.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:19.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:19.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:19.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:19.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2026-02-22T20:21:19.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:19.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:19.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:19.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:19.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:19.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:19.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:19.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:19.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:19.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:19.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:19.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:19.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:19.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:19.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:19.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:19.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:19.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:19.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:19.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:19.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:19.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:19.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:19.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:19.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:19.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:19.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2026-02-22T20:21:19.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:19.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_32 2026-02-22T20:21:19.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:19.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '' 2026-02-22T20:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:19.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:19.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:19.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:19.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:19.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:19.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2026-02-22T20:21:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2026-02-22T20:21:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:19.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:19.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:19.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:19.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:19.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:19.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:19.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:19.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:19.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:19.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=14594], [object_number=13, tag_tid=1, entry_tid=14593], [object_number=12, tag_tid=1, entry_tid=14592], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:19.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:19.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:19.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:19.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=14594' '!=' 'object_number=14, tag_tid=1, entry_tid=14594' 2026-02-22T20:21:19.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:19.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:21:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_32 2026-02-22T20:21:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_32 ']' 2026-02-22T20:21:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_32' 2026-02-22T20:21:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2026-02-22T20:21:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:21.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:21.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:21.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:21.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:21.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2026-02-22T20:21:22.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:22.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_32 2026-02-22T20:21:22.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:22.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '' 2026-02-22T20:21:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:22.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:22.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:22.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:22.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:22.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:22.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:22.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:22.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:22.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:22.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2026-02-22T20:21:22.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:22.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:22.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:22.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2026-02-22T20:21:22.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:22.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:22.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:21:22.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:22.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_32.status 2026-02-22T20:21:22.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_32.status 2026-02-22T20:21:22.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:22.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:22.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2026-02-22T20:21:22.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:21:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:21:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:22.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=14594], [object_number=15, tag_tid=1, entry_tid=14591]]], state=connected] 2026-02-22T20:21:22.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.status 2026-02-22T20:21:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:21:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2026-02-22T20:21:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:22.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:21:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:23.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2026-02-22T20:21:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_32': 2026-02-22T20:21:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11f8108ba84a 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11f8108ba84a 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:11:49 2026 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:11:49 2026 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:12:57 2026 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11f8108ba84a 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 907d91bf-8f4f-45e7-b86a-8581f1a3ba6d 2026-02-22T20:21:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:21:23.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:23.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_32 snap 2026-02-22T20:21:23.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:23.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:21:23.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:23.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:21:23.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.export 2026-02-22T20:21:23.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_32.export 2026-02-22T20:21:23.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_32.export 2026-02-22T20:21:23.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.export 2026-02-22T20:21:23.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:23.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_32.export 2026-02-22T20:21:23.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:23.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_32.export 2026-02-22T20:21:23.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_32.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_32.export 2026-02-22T20:21:23.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:23.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:21:23.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_33 2026-02-22T20:21:23.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_33 snap 2026-02-22T20:21:23.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:23.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:23.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:21:23.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:23.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:21:24.454 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T20:21:24.445+0000 7fdfce879640 -1 reset not still connected to 0x555a9c4b5ee0 2026-02-22T20:21:24.455 INFO:tasks.ceph.cluster2.osd.1.trial066.stderr:2026-02-22T20:21:24.445+0000 7fdfce879640 -1 reset not still connected to 0x555aa1a696c0 2026-02-22T20:21:24.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2026-02-22T20:21:24.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:24.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:24.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:21:24.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2026-02-22T20:21:24.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:24.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:24.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:21:24.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:21:24.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:24.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:24.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2026-02-22T20:21:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:21:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:21:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:21:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:21:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:25.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:21:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:25.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:25.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:25.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:25.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2026-02-22T20:21:25.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2026-02-22T20:21:25.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:25.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:25.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:25.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2026-02-22T20:21:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:21:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:25.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:21:25.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:25.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:25.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:25.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:25.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:25.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:25.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:25.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:21:25.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:21:25.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2026-02-22T20:21:25.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2026-02-22T20:21:25.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:25.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:25.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:25.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2026-02-22T20:21:25.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:21:25.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:25.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:25.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:25.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:25.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:25.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:25.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:21:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2026-02-22T20:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2026-02-22T20:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:26.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:26.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2026-02-22T20:21:26.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:21:26.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:26.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:21:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:21:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:26.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2026-02-22T20:21:26.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:21:26.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:21:26.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:21:26.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:21:26.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:21:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:21:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:21:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2026-02-22T20:21:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_33 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_33 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_33 ']' 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_33' 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2026-02-22T20:21:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:26.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:26.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2026-02-22T20:21:26.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2026-02-22T20:21:26.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:26.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:26.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:26.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:26.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:26.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:26.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2026-02-22T20:21:26.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2026-02-22T20:21:26.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:26.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2026-02-22T20:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:26.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:26.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:26.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:26.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:26.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:26.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:26.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:26.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2026-02-22T20:21:26.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2026-02-22T20:21:26.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:26.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:26.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:26.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:26.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:26.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:26.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:26.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:26.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:26.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:26.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:26.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:26.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:26.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:26.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2026-02-22T20:21:26.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:26.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_33 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_33 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_33 '' 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_33 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_33.status 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_33.status 2026-02-22T20:21:26.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:26.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.status 2026-02-22T20:21:26.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2026-02-22T20:21:26.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.status 2026-02-22T20:21:26.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:21:26.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:21:26.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:26.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11239], [object_number=10, tag_tid=1, entry_tid=11238]]], state=connected] 2026-02-22T20:21:26.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11239], [object_number=10, tag_tid=1, entry_tid=11238]]], state=connected] 2026-02-22T20:21:26.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.status 2026-02-22T20:21:26.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:21:26.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2026-02-22T20:21:26.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:26.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:26.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_33 2026-02-22T20:21:26.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2026-02-22T20:21:26.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:26.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:26.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_33 2026-02-22T20:21:26.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:26.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_33.status 2026-02-22T20:21:26.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_33.status 2026-02-22T20:21:26.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:26.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.status 2026-02-22T20:21:26.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2026-02-22T20:21:26.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.status 2026-02-22T20:21:26.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:21:26.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:21:26.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:26.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11239], [object_number=10, tag_tid=1, entry_tid=11238]]], state=connected] 2026-02-22T20:21:26.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11239], [object_number=10, tag_tid=1, entry_tid=11238]]], state=connected] 2026-02-22T20:21:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.status 2026-02-22T20:21:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:21:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:21:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2026-02-22T20:21:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:21:26.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:26.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:26.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:21:26.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:27.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2026-02-22T20:21:27.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_33': 2026-02-22T20:21:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:21:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:21:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:21:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aabeb0bb65 2026-02-22T20:21:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aabeb0bb65 2026-02-22T20:21:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:21:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:21:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:21:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:21:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:11:55 2026 2026-02-22T20:21:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:11:55 2026 2026-02-22T20:21:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:11:55 2026 2026-02-22T20:21:27.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:21:27.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aabeb0bb65 2026-02-22T20:21:27.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:21:27.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:21:27.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: e45c3de2-2b0c-482a-aa57-412779f2b406 2026-02-22T20:21:27.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:21:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_33 snap 2026-02-22T20:21:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:21:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:21:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.export 2026-02-22T20:21:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_33.export 2026-02-22T20:21:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_33.export 2026-02-22T20:21:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.export 2026-02-22T20:21:28.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_33.export 2026-02-22T20:21:28.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:28.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_33.export 2026-02-22T20:21:28.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_33.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_33.export 2026-02-22T20:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_34 2026-02-22T20:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_34 snap 2026-02-22T20:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:30.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2026-02-22T20:21:30.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:30.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:30.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:30.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:21:30.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:21:30.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:21:30.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2026-02-22T20:21:30.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:21:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:21:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:21:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:21:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:21:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:30.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:30.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:21:30.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:30.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:30.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:30.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:30.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2026-02-22T20:21:30.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:30.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:30.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:30.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:30.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:30.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:21:30.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:21:30.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:30.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:30.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:30.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:30.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2026-02-22T20:21:30.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:30.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:30.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:30.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:30.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2026-02-22T20:21:30.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:21:30.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:30.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:21:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:21:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2026-02-22T20:21:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:21:31.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:21:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:21:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:21:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:21:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:21:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:21:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2026-02-22T20:21:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_34 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:31.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_34 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_34 ']' 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_34' 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:31.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:31.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:31.357 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:31.358 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:31.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:31.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:31.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:31.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:31.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:31.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:31.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:31.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:31.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:31.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:31.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:31.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:31.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:31.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:31.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_34 2026-02-22T20:21:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '' 2026-02-22T20:21:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:31.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:31.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:31.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:31.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:31.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=1, entry_tid=18905], [object_number=16, tag_tid=1, entry_tid=18904], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:31.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:31.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2026-02-22T20:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:31.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2026-02-22T20:21:31.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:31.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:31.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:31.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:31.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:31.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:31.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:31.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:31.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:31.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:31.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:31.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:31.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:31.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:31.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=1, entry_tid=18905], [object_number=16, tag_tid=1, entry_tid=18904], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=1, entry_tid=18905' '!=' '' 2026-02-22T20:21:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:21:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=1, entry_tid=18905' = '' 2026-02-22T20:21:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:31.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:21:31.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:31.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:31.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:21:31.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:31.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:31.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:31.580 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:21:31.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:21:31.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:31.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:31.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:21:31.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:31.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:31.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:31.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=18905 2026-02-22T20:21:31.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 18905 2026-02-22T20:21:31.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:31.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:31.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_34 2026-02-22T20:21:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_34 ']' 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_34' 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:31.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:31.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:31.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2026-02-22T20:21:31.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:31.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:31.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:31.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:31.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:31.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:31.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:31.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2026-02-22T20:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:31.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:31.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:31.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:31.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:31.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:31.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:31.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2026-02-22T20:21:31.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:31.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:31.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:31.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:31.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:32.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_34 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '' 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:32.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:32.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:32.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:32.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:32.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=1, entry_tid=18905], [object_number=16, tag_tid=1, entry_tid=18904], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:32.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2026-02-22T20:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2026-02-22T20:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:32.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:32.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:32.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:32.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:32.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:32.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:32.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:32.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=1, entry_tid=18905], [object_number=16, tag_tid=1, entry_tid=18904], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=1, entry_tid=18905' '!=' 'object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:21:32.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_34 2026-02-22T20:21:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_34 ']' 2026-02-22T20:21:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_34' 2026-02-22T20:21:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2026-02-22T20:21:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:32.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:32.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:32.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:32.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:32.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:32.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:32.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:32.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:32.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:32.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:32.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:32.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:32.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:32.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2026-02-22T20:21:32.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:32.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:32.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:32.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:32.614 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:32.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:32.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2026-02-22T20:21:32.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:32.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:32.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:32.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:32.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:32.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:32.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:32.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:32.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_34 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '' 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:32.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:32.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:32.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=1, entry_tid=18905], [object_number=16, tag_tid=1, entry_tid=18904], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:32.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:21:32.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2026-02-22T20:21:32.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:32.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:32.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:32.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2026-02-22T20:21:32.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:32.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:32.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:32.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:32.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:32.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:32.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:32.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:32.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:32.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:32.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:32.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:32.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=1, entry_tid=18905], [object_number=16, tag_tid=1, entry_tid=18904], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:32.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:32.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:32.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:32.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=1, entry_tid=18905' '!=' 'object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:32.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:32.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_34 2026-02-22T20:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_34 ']' 2026-02-22T20:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_34' 2026-02-22T20:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2026-02-22T20:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:33.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:33.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:33.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:33.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:33.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:33.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:33.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2026-02-22T20:21:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:33.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:33.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:33.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:33.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:33.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:33.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:33.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:33.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:33.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:33.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:33.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:33.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_34 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '' 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:33.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:33.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:33.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:33.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:33.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:33.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:33.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=1, entry_tid=18905], [object_number=16, tag_tid=1, entry_tid=18904], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:33.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:33.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:21:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2026-02-22T20:21:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2026-02-22T20:21:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:33.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:33.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:33.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:33.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:33.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:33.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:33.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:33.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=1, entry_tid=18905], [object_number=16, tag_tid=1, entry_tid=18904], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:33.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:33.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:33.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:33.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=17, tag_tid=1, entry_tid=18905' '!=' 'object_number=17, tag_tid=1, entry_tid=18905' 2026-02-22T20:21:33.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:33.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:21:35.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_34 2026-02-22T20:21:35.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:35.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_34 ']' 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_34' 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:35.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:35.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:35.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:35.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:35.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:35.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:35.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:35.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:35.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:35.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:35.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:35.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:35.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:35.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:35.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2026-02-22T20:21:35.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:35.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:35.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:35.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:35.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:35.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:35.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:35.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:35.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:35.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:35.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-22T20:21:35.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:35.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:35.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:35.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2026-02-22T20:21:35.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:35.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:35.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:35.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:35.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:35.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:35.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:35.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:35.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:35.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2026-02-22T20:21:35.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:35.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_34 2026-02-22T20:21:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '' 2026-02-22T20:21:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:35.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:35.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:35.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:35.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:35.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:35.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:35.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:35.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:35.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:21:35.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2026-02-22T20:21:35.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2026-02-22T20:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_34.status 2026-02-22T20:21:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_34.status 2026-02-22T20:21:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2026-02-22T20:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:35.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:21:35.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:21:35.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:35.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:35.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=18903], [object_number=18, tag_tid=1, entry_tid=18902]]], state=connected] 2026-02-22T20:21:35.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.status 2026-02-22T20:21:35.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:21:35.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:21:35.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:35.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2026-02-22T20:21:35.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:35.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:35.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:35.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:35.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:35.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:21:35.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:36.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2026-02-22T20:21:37.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_34': 2026-02-22T20:21:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:21:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:21:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:21:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1127ed8ac064 2026-02-22T20:21:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1127ed8ac064 2026-02-22T20:21:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:21:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:21:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:21:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:12:00 2026 2026-02-22T20:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:12:00 2026 2026-02-22T20:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:12:00 2026 2026-02-22T20:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1127ed8ac064 2026-02-22T20:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 77872b5d-5e17-4d60-86dd-e92d66e1f2fd 2026-02-22T20:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_34 snap 2026-02-22T20:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.export 2026-02-22T20:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_34.export 2026-02-22T20:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_34.export 2026-02-22T20:21:37.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.export 2026-02-22T20:21:37.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_34.export 2026-02-22T20:21:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:37.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_34.export 2026-02-22T20:21:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_34.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_34.export 2026-02-22T20:21:37.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:37.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:21:37.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_35 2026-02-22T20:21:37.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_35 snap 2026-02-22T20:21:37.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:37.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:37.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:37.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:37.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:21:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2026-02-22T20:21:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2026-02-22T20:21:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:21:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:38.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:38.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:39.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2026-02-22T20:21:39.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:39.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:39.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:39.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:21:39.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:21:39.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:21:39.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2026-02-22T20:21:39.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:21:39.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:21:39.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:39.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:39.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:21:39.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:21:39.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:39.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:21:39.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:39.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:21:39.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:39.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2026-02-22T20:21:39.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:21:39.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:39.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:39.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:39.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:39.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:21:39.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:21:39.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:39.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:39.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:39.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:39.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2026-02-22T20:21:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2026-02-22T20:21:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:21:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:21:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:21:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:39.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:39.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:39.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:39.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2026-02-22T20:21:39.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:39.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:40.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2026-02-22T20:21:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:21:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:40.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:40.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:21:40.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:21:40.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:40.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:40.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:40.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:40.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2026-02-22T20:21:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:40.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:40.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_35 2026-02-22T20:21:40.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:21:40.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:40.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:40.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:40.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:40.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:40.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:21:40.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:21:40.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:40.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:40.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:40.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:40.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_35 2026-02-22T20:21:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:21:40.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:21:40.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:21:40.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:21:40.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:21:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:21:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:21:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2026-02-22T20:21:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_35 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_35 2026-02-22T20:21:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_35 ']' 2026-02-22T20:21:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_35' 2026-02-22T20:21:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2026-02-22T20:21:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:40.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:40.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:40.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:40.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:40.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:40.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:40.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:40.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:40.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:40.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2026-02-22T20:21:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:40.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:40.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:40.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:40.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:40.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:40.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:40.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:40.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_35 2026-02-22T20:21:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:40.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:40.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:40.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:40.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:40.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:40.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:40.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:40.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:40.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:40.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:40.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_35 2026-02-22T20:21:40.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:40.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:40.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:40.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:40.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:40.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:40.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:40.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:40.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:40.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:40.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:40.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:40.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:40.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:40.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:40.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:40.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:40.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:40.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:40.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:40.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:40.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:40.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:40.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:40.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:40.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:40.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:40.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:40.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:40.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:40.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:40.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_35 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_35 '' 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:40.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:40.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_35.status 2026-02-22T20:21:40.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_35.status 2026-02-22T20:21:40.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:40.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:40.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2026-02-22T20:21:40.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:40.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:40.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:40.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:40.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:40.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30772], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770], [object_number=29, tag_tid=1, entry_tid=30769]]], state=connected] 2026-02-22T20:21:40.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:40.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2026-02-22T20:21:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2026-02-22T20:21:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_35.status 2026-02-22T20:21:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_35.status 2026-02-22T20:21:40.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2026-02-22T20:21:40.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30772], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770], [object_number=29, tag_tid=1, entry_tid=30769]]], state=connected] 2026-02-22T20:21:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=28, tag_tid=1, entry_tid=30772' '!=' '' 2026-02-22T20:21:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:21:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=28, tag_tid=1, entry_tid=30772' = '' 2026-02-22T20:21:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:21:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:21:40.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:40.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:40.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:40.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:21:40.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30772 2026-02-22T20:21:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 30772 2026-02-22T20:21:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:40.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_35 2026-02-22T20:21:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_35 ']' 2026-02-22T20:21:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_35' 2026-02-22T20:21:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2026-02-22T20:21:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2026-02-22T20:21:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:40.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:40.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:40.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:40.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:40.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:40.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:40.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:40.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:40.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:40.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:40.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:40.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:40.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:40.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:40.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:40.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:40.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:40.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:40.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:40.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:40.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:40.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:40.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:40.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:41.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:41.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:41.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:41.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:41.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2026-02-22T20:21:41.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:41.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:41.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:41.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:41.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:41.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:41.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:41.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_35 2026-02-22T20:21:41.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:41.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:41.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:41.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:41.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:41.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:41.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:41.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:41.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:41.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_35 2026-02-22T20:21:41.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:41.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:41.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:41.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:41.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:41.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:41.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:41.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:41.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:41.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_35 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_35 '' 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_35.status 2026-02-22T20:21:41.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_35.status 2026-02-22T20:21:41.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:41.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:41.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2026-02-22T20:21:41.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:41.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:41.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:41.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:41.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:41.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30772], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770], [object_number=29, tag_tid=1, entry_tid=30769]]], state=connected] 2026-02-22T20:21:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:41.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:41.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2026-02-22T20:21:41.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:41.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:41.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:41.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2026-02-22T20:21:41.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:41.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:41.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:41.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:41.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_35.status 2026-02-22T20:21:41.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_35.status 2026-02-22T20:21:41.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:41.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:41.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2026-02-22T20:21:41.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:41.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:41.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:41.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:41.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:41.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30772], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770], [object_number=29, tag_tid=1, entry_tid=30769]]], state=connected] 2026-02-22T20:21:41.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:41.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:41.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:41.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=28, tag_tid=1, entry_tid=30772' '!=' 'object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:41.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:41.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:21:41.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_35 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_35 ']' 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_35' 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:41.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:41.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:41.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:41.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:41.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:41.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:41.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:41.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:41.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:41.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:41.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:41.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2026-02-22T20:21:41.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:41.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:41.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:41.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:41.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:41.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:41.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:41.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:41.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:41.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:41.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:41.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:41.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:41.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_35 2026-02-22T20:21:41.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:41.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:41.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:41.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:41.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:41.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:41.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:41.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:41.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:41.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:41.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_35 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:41.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:41.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:41.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_35 2026-02-22T20:21:41.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_35 '' 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_35.status 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_35.status 2026-02-22T20:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2026-02-22T20:21:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:42.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:42.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:42.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:42.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:42.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30772], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770], [object_number=29, tag_tid=1, entry_tid=30769]]], state=connected] 2026-02-22T20:21:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:42.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2026-02-22T20:21:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:42.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:42.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:42.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2026-02-22T20:21:42.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:42.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:42.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:42.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:42.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_35.status 2026-02-22T20:21:42.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_35.status 2026-02-22T20:21:42.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:42.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:42.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2026-02-22T20:21:42.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:42.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:42.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:42.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:42.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:42.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30772], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770], [object_number=29, tag_tid=1, entry_tid=30769]]], state=connected] 2026-02-22T20:21:42.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:42.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:42.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:42.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=28, tag_tid=1, entry_tid=30772' '!=' 'object_number=28, tag_tid=1, entry_tid=30772' 2026-02-22T20:21:42.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:42.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_35 2026-02-22T20:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_35 ']' 2026-02-22T20:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_35' 2026-02-22T20:21:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2026-02-22T20:21:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:42.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:42.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:42.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:42.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2026-02-22T20:21:42.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:42.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:42.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:42.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:42.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:42.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:42.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:42.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:42.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:42.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:42.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:42.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:42.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:42.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:42.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:43.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:43.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_35 2026-02-22T20:21:43.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:43.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:43.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:43.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:43.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:43.032 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:43.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-22T20:21:43.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:43.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:43.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:43.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_35 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:43.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:43.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:43.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:43.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_35 2026-02-22T20:21:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_35 2026-02-22T20:21:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_35 '' 2026-02-22T20:21:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:43.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:43.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_35.status 2026-02-22T20:21:43.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_35.status 2026-02-22T20:21:43.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2026-02-22T20:21:43.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:43.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:43.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:43.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:43.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:43.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:43.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:43.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:43.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2026-02-22T20:21:43.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:43.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:43.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:43.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2026-02-22T20:21:43.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:43.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:43.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:21:43.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:43.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_35.status 2026-02-22T20:21:43.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_35.status 2026-02-22T20:21:43.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:43.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:43.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2026-02-22T20:21:43.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:21:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:21:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30771], [object_number=30, tag_tid=1, entry_tid=30770]]], state=connected] 2026-02-22T20:21:43.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.status 2026-02-22T20:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:21:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:44.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2026-02-22T20:21:44.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_35': 2026-02-22T20:21:44.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:21:44.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:21:44.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:21:44.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e3c9b864c5 2026-02-22T20:21:44.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e3c9b864c5 2026-02-22T20:21:44.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:12:07 2026 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:12:07 2026 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:13:08 2026 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e3c9b864c5 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:21:44.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 95d434ae-767e-4a81-ae85-af82c746aac1 2026-02-22T20:21:44.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:21:44.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:44.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_35 snap 2026-02-22T20:21:44.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:44.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:21:44.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:44.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:21:44.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.export 2026-02-22T20:21:44.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_35.export 2026-02-22T20:21:44.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_35.export 2026-02-22T20:21:44.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.export 2026-02-22T20:21:44.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_35.export 2026-02-22T20:21:44.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_35.export 2026-02-22T20:21:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_35.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_35.export 2026-02-22T20:21:44.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:44.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:21:44.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_36 2026-02-22T20:21:44.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_36 snap 2026-02-22T20:21:44.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:44.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:44.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:44.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:44.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:21:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2026-02-22T20:21:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2026-02-22T20:21:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:21:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:47.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:48.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2026-02-22T20:21:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:21:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:21:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:21:48.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2026-02-22T20:21:48.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:21:48.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:21:48.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:48.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:48.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:21:48.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:21:48.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:48.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:21:48.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:21:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2026-02-22T20:21:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:21:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:48.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:48.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:48.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:21:48.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:48.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:48.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:48.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:21:48.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:48.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:48.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:48.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:48.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:48.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:48.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:48.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:21:48.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:21:48.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:48.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:48.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:48.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:48.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2026-02-22T20:21:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2026-02-22T20:21:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:21:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:21:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:21:48.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2026-02-22T20:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2026-02-22T20:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:21:49.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:21:49.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2026-02-22T20:21:49.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:49.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2026-02-22T20:21:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:21:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:49.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:21:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:21:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:21:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:21:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:49.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:49.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:49.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:49.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2026-02-22T20:21:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:21:49.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:21:49.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:21:49.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:21:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:21:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:21:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:21:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2026-02-22T20:21:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_36 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:49.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_36 2026-02-22T20:21:49.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:49.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_36 ']' 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_36' 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:49.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:49.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:49.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:49.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:49.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:49.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:49.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:49.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:49.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:49.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:49.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:49.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:49.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:49.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2026-02-22T20:21:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:49.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:49.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:49.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:49.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_36 2026-02-22T20:21:49.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:49.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:49.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:49.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:49.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:49.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:49.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:49.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:49.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:49.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:49.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:49.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_36 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:49.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:49.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:49.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:49.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:49.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:49.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:49.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_36 2026-02-22T20:21:49.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:49.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:49.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '' 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:49.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:49.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:49.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:49.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=40998], [object_number=37, tag_tid=1, entry_tid=40997], [object_number=36, tag_tid=1, entry_tid=40996], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:49.696 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:49.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:21:49.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2026-02-22T20:21:49.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:49.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:49.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:49.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2026-02-22T20:21:49.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:49.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:49.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:49.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:49.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:49.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:49.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:49.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:49.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:49.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:49.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=40998], [object_number=37, tag_tid=1, entry_tid=40997], [object_number=36, tag_tid=1, entry_tid=40996], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:49.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:49.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:49.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:49.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=40998' '!=' '' 2026-02-22T20:21:49.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:21:49.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=40998' = '' 2026-02-22T20:21:49.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:49.732 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:21:49.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:49.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:49.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:21:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:49.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:21:49.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:21:49.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:49.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:49.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:21:49.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:49.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:49.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=40998 2026-02-22T20:21:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 40998 2026-02-22T20:21:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_36 2026-02-22T20:21:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:49.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_36 ']' 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_36' 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:49.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:49.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:49.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:49.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:50.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:50.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:50.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:50.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_36 2026-02-22T20:21:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:50.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:50.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:50.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:50.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:50.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:50.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:50.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:50.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:50.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:50.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:50.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:50.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:50.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:50.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:50.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_36 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:50.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:50.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:50.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:50.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:50.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:50.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:50.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:50.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:50.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:50.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:50.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:50.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:50.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:50.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:50.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:50.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:50.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:50.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:50.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_36 2026-02-22T20:21:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '' 2026-02-22T20:21:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:50.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:50.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:50.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:50.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:50.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:50.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:50.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:50.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:50.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:50.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:50.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=40998], [object_number=37, tag_tid=1, entry_tid=40997], [object_number=36, tag_tid=1, entry_tid=40996], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:50.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:50.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:21:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2026-02-22T20:21:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2026-02-22T20:21:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:50.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:50.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:50.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:50.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:50.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:50.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:50.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=40998], [object_number=37, tag_tid=1, entry_tid=40997], [object_number=36, tag_tid=1, entry_tid=40996], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:50.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:50.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:50.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:50.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=40998' '!=' 'object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:50.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:50.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:21:50.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_36 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_36 ']' 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_36' 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:50.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:50.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:50.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:50.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:50.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:50.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:50.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:50.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:50.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2026-02-22T20:21:50.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:50.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:50.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:50.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:50.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:50.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:50.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:50.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:50.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_36 2026-02-22T20:21:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:50.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:50.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:50.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:50.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:50.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_36 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:50.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:50.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:50.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:50.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:50.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:50.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_36 2026-02-22T20:21:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:50.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:50.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '' 2026-02-22T20:21:50.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:50.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:50.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:50.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:50.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:50.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:50.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:50.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:50.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:50.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:51.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:51.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:51.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:51.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:51.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=40998], [object_number=37, tag_tid=1, entry_tid=40997], [object_number=36, tag_tid=1, entry_tid=40996], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:51.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:51.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:21:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2026-02-22T20:21:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2026-02-22T20:21:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:51.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:51.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:51.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=40998], [object_number=37, tag_tid=1, entry_tid=40997], [object_number=36, tag_tid=1, entry_tid=40996], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:51.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:51.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:51.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:51.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=40998' '!=' 'object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:51.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:51.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:21:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_36 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_36 ']' 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_36' 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:51.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:51.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:51.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:51.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:51.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:51.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:51.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:51.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:51.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:51.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:51.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:51.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_36 2026-02-22T20:21:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:52.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:52.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:52.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:52.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:52.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:52.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:52.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_36 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:52.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:52.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:52.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:52.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_36 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '' 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:52.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:52.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:52.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:52.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:52.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:52.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=40998], [object_number=37, tag_tid=1, entry_tid=40997], [object_number=36, tag_tid=1, entry_tid=40996], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:52.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:52.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:52.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:52.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:52.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:52.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:52.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:52.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:52.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:52.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:52.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:52.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:52.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=40998], [object_number=37, tag_tid=1, entry_tid=40997], [object_number=36, tag_tid=1, entry_tid=40996], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:52.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:52.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:52.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=40998' '!=' 'object_number=38, tag_tid=1, entry_tid=40998' 2026-02-22T20:21:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:52.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:21:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_36 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_36 ']' 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_36' 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:53.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:53.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:53.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:53.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:53.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:53.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:53.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2026-02-22T20:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:53.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:53.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:53.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:53.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:53.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:53.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:53.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2026-02-22T20:21:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:53.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:53.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:53.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:53.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:53.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:53.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_36 2026-02-22T20:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:54.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-22T20:21:54.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:54.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:54.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:54.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_36 2026-02-22T20:21:54.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:21:54.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:21:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:21:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:21:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:54.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:54.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:21:54.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_36 2026-02-22T20:21:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_36 2026-02-22T20:21:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '' 2026-02-22T20:21:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:54.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:54.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:54.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:54.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:54.131 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:54.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:54.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:54.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:54.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:21:54.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2026-02-22T20:21:54.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:54.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:54.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:54.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2026-02-22T20:21:54.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:54.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:54.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:21:54.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:54.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_36.status 2026-02-22T20:21:54.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_36.status 2026-02-22T20:21:54.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:54.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:54.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2026-02-22T20:21:54.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:54.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:21:54.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:21:54.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:54.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:54.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=40998], [object_number=39, tag_tid=1, entry_tid=40995]]], state=connected] 2026-02-22T20:21:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.status 2026-02-22T20:21:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:21:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:21:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2026-02-22T20:21:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:21:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:55.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2026-02-22T20:21:55.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_36': 2026-02-22T20:21:55.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:21:55.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:21:55.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11f89efe647c 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11f89efe647c 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:12:17 2026 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:12:17 2026 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:14:19 2026 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11f89efe647c 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 87e56335-e8fa-432e-a65a-ec25fedacd1b 2026-02-22T20:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_36 snap 2026-02-22T20:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.export 2026-02-22T20:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_36.export 2026-02-22T20:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_36.export 2026-02-22T20:21:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.export 2026-02-22T20:21:55.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:55.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_36.export 2026-02-22T20:21:55.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:21:55.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_36.export 2026-02-22T20:21:55.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_36.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_36.export 2026-02-22T20:21:55.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:55.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:21:55.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_37 2026-02-22T20:21:55.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_37 snap 2026-02-22T20:21:55.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:55.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:55.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:21:55.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:55.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:21:56.541 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2026-02-22T20:21:56.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2026-02-22T20:21:56.542 INFO:tasks.rbd_mirror_thrash:waiting for 2.489630302255864 secs before reviving daemons 2026-02-22T20:21:56.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:21:56.530+0000 7f53158ea640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 87305) UID: 1000 2026-02-22T20:21:56.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:21:56.554+0000 7f530e8dc640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x559d28d0ca80 handle_process_entry_safe: failed to commit journal event: (85) Interrupted system call should be restarted 2026-02-22T20:21:56.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:21:56.554+0000 7f530e8dc640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x559d28d0ca80 handle_replay_complete: replay encountered an error: (85) Interrupted system call should be restarted 2026-02-22T20:21:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2026-02-22T20:21:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:21:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2026-02-22T20:21:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:21:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:21:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:21:57.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2026-02-22T20:21:57.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:57.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:57.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:21:57.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:21:57.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:21:57.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:21:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:21:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2026-02-22T20:21:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:21:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:21:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:21:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:21:57.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:57.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:21:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2026-02-22T20:21:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:21:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2026-02-22T20:21:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:21:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:21:57.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:21:57.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:57.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:57.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:57.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:57.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2026-02-22T20:21:57.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2026-02-22T20:21:57.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:21:57.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2026-02-22T20:21:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:21:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:21:57.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:21:57.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:58.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:58.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:58.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:58.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2026-02-22T20:21:58.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:21:58.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:21:58.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:21:58.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:21:58.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:21:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:21:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:21:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2026-02-22T20:21:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:58.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:21:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:21:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_37 2026-02-22T20:21:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:21:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:21:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:21:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:21:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:21:58.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:58.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:58.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_37 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_37 ']' 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_37' 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:58.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:58.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:58.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:58.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:58.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:58.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2026-02-22T20:21:58.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:21:58.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:58.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2026-02-22T20:21:58.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:58.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:58.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:58.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:58.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:58.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2026-02-22T20:21:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2026-02-22T20:21:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:58.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:58.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:58.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_37 2026-02-22T20:21:58.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:58.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:58.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:58.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:58.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:58.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:58.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:58.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:58.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:58.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:58.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:58.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_37 2026-02-22T20:21:58.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_37 2026-02-22T20:21:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:21:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_37 '' 2026-02-22T20:21:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:58.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_37.status 2026-02-22T20:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_37.status 2026-02-22T20:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:58.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2026-02-22T20:21:58.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:58.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:21:58.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:21:58.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:58.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=27863], [object_number=26, tag_tid=1, entry_tid=27862]]], state=connected] 2026-02-22T20:21:58.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=24, tag_tid=1, entry_tid=27864], [object_number=27, tag_tid=1, entry_tid=27863], [object_number=26, tag_tid=1, entry_tid=27862], [object_number=25, tag_tid=1, entry_tid=27861]]], state=connected] 2026-02-22T20:21:58.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:58.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:21:58.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2026-02-22T20:21:58.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:58.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:58.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:21:58.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2026-02-22T20:21:58.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:58.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:58.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:21:58.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:58.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_37.status 2026-02-22T20:21:58.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_37.status 2026-02-22T20:21:58.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:58.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:58.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2026-02-22T20:21:58.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:58.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:21:58.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:21:58.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:58.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=27863], [object_number=26, tag_tid=1, entry_tid=27862]]], state=connected] 2026-02-22T20:21:58.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=24, tag_tid=1, entry_tid=27864], [object_number=27, tag_tid=1, entry_tid=27863], [object_number=26, tag_tid=1, entry_tid=27862], [object_number=25, tag_tid=1, entry_tid=27861]]], state=connected] 2026-02-22T20:21:58.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:58.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=24, tag_tid=1, entry_tid=27864' 2026-02-22T20:21:58.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=24, tag_tid=1, entry_tid=27864' 2026-02-22T20:21:58.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=24, tag_tid=1, entry_tid=27864' '!=' '' 2026-02-22T20:21:58.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:21:58.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=24, tag_tid=1, entry_tid=27864' = '' 2026-02-22T20:21:58.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=24, tag_tid=1, entry_tid=27864' 2026-02-22T20:21:58.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:21:58.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:58.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:58.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:21:58.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=24, tag_tid=1, entry_tid=27864' 2026-02-22T20:21:58.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:21:58.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:58.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:21:58.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:21:58.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:58.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:58.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:21:58.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=24, tag_tid=1, entry_tid=27864' 2026-02-22T20:21:58.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:21:58.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:21:58.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=27864 2026-02-22T20:21:58.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 27864 2026-02-22T20:21:58.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:21:58.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:21:58.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:21:58.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_37 2026-02-22T20:21:58.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:58.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_37 ']' 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_37' 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:21:58.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:21:58.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:21:58.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:58.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2026-02-22T20:21:58.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:58.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:21:58.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:21:58.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:58.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:58.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:21:58.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:21:58.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:21:58.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:21:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2026-02-22T20:21:58.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-22T20:21:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2026-02-22T20:21:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:21:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:21:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:21:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:21:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:21:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:58.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:58.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:21:58.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2026-02-22T20:21:58.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2026-02-22T20:21:58.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:21:58.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_37 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:21:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:21:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:21:58.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:21:58.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:58.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:58.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:21:58.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_37 2026-02-22T20:21:59.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_37 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_37 '' 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:21:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_37.status 2026-02-22T20:21:59.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_37.status 2026-02-22T20:21:59.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:59.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:59.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2026-02-22T20:21:59.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:59.032 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2026-02-22T20:21:59.032 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:21:59.032 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2026-02-22T20:21:59.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-22T20:21:59.032 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-22T20:21:59.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-22T20:21:59.034 INFO:tasks.rbd_mirror_thrash:waiting for 72.35818910395778 secs before thrashing 2026-02-22T20:21:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:21:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:21:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=27863], [object_number=26, tag_tid=1, entry_tid=27862]]], state=connected] 2026-02-22T20:21:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=27863], [object_number=26, tag_tid=1, entry_tid=27862]]], state=connected] 2026-02-22T20:21:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:59.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:21:59.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_37.status 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_37.status 2026-02-22T20:21:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:21:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2026-02-22T20:21:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:59.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:21:59.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:21:59.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:21:59.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=27863], [object_number=26, tag_tid=1, entry_tid=27862]]], state=connected] 2026-02-22T20:21:59.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=27863], [object_number=26, tag_tid=1, entry_tid=27862]]], state=connected] 2026-02-22T20:21:59.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.status 2026-02-22T20:21:59.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:21:59.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:21:59.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:21:59.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2026-02-22T20:21:59.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:21:59.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:21:59.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:21:59.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:21:59.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:21:59.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:21:59.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:22:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2026-02-22T20:22:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_37': 2026-02-22T20:22:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aaa5b75ca9 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aaa5b75ca9 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:12:30 2026 2026-02-22T20:22:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:12:30 2026 2026-02-22T20:22:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:13:31 2026 2026-02-22T20:22:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:22:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aaa5b75ca9 2026-02-22T20:22:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:22:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:22:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 29e6855c-aff8-43eb-b6d8-03f105c7f8c6 2026-02-22T20:22:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_37 snap 2026-02-22T20:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.export 2026-02-22T20:22:00.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_37.export 2026-02-22T20:22:00.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_37.export 2026-02-22T20:22:00.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.export 2026-02-22T20:22:00.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:22:00.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_37.export 2026-02-22T20:22:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:22:00.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_37.export 2026-02-22T20:22:00.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_37.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_37.export 2026-02-22T20:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_38 2026-02-22T20:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_38 snap 2026-02-22T20:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:22:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2026-02-22T20:22:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2026-02-22T20:22:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:01.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:22:01.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:22:01.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:01.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2026-02-22T20:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2026-02-22T20:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:22:02.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:02.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:22:02.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2026-02-22T20:22:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:02.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:22:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:22:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:02.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2026-02-22T20:22:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:03.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:22:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:03.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:03.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:03.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:03.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2026-02-22T20:22:03.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:03.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:03.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:22:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:03.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:03.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2026-02-22T20:22:03.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:03.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 1 2026-02-22T20:22:04.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:04.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:22:04.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:04.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2026-02-22T20:22:04.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:22:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:22:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:04.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:04.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:04.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:04.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2026-02-22T20:22:04.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:04.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:04.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:04.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2026-02-22T20:22:04.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:04.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:04.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:04.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:04.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:04.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:22:04.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:22:04.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:04.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:04.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:04.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:04.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2026-02-22T20:22:04.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:04.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:04.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:04.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:22:04.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:04.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:04.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:04.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:04.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:04.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:04.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:04.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:22:04.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:22:04.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:04.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:04.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:04.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2026-02-22T20:22:04.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:04.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:04.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2026-02-22T20:22:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:22:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:04.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:04.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:04.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:04.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:04.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:22:04.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:22:04.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:04.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:04.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:04.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:04.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2026-02-22T20:22:04.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:04.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:04.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 2 2026-02-22T20:22:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:06.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:06.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:06.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:06.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2026-02-22T20:22:06.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:06.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:06.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:06.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:06.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2026-02-22T20:22:06.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:22:06.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:06.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:06.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:06.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:06.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:22:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:06.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:06.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:06.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:06.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2026-02-22T20:22:06.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:06.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:06.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:06.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:22:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:22:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:06.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:06.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:06.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:06.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2026-02-22T20:22:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:06.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:06.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:22:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:22:06.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2026-02-22T20:22:06.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:06.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:06.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:06.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:06.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 4 2026-02-22T20:22:07.230 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4720029276663187) > weight (0.25) 2026-02-22T20:22:07.230 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2026-02-22T20:22:07.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2026-02-22T20:22:07.231 INFO:tasks.rbd_mirror_thrash:waiting for 7.9444726268620585 secs before reviving daemons 2026-02-22T20:22:07.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:22:07.222+0000 7f94debd6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 88028) UID: 1000 2026-02-22T20:22:10.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:22:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2026-02-22T20:22:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:22:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:22:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:22:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:22:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:22:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:10.750 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:10.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2026-02-22T20:22:10.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:10.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:10.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2026-02-22T20:22:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:22:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:10.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:22:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:10.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2026-02-22T20:22:10.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:10.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:10.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:10.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:22:10.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2026-02-22T20:22:10.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:10.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:10.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:10.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:10.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:10.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:22:10.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:22:10.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:10.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2026-02-22T20:22:11.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:11.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:11.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:11.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:11.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 8 2026-02-22T20:22:15.176 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2026-02-22T20:22:15.176 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:22:15.176 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2026-02-22T20:22:15.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-22T20:22:15.177 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-22T20:22:15.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-22T20:22:15.179 INFO:tasks.rbd_mirror_thrash:waiting for 100.44211596635265 secs before thrashing 2026-02-22T20:22:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:22:19.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:19.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2026-02-22T20:22:19.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:22:19.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:19.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:22:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:22:19.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2026-02-22T20:22:19.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:19.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:19.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2026-02-22T20:22:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:22:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:19.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:19.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:22:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:19.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:19.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:19.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:19.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2026-02-22T20:22:19.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:19.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:19.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:19.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:22:19.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2026-02-22T20:22:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:19.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:19.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2026-02-22T20:22:19.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:22:19.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:22:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:22:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:19.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:19.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:19.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:19.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2026-02-22T20:22:19.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:19.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:19.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:19.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:19.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 8 2026-02-22T20:22:26.217 INFO:tasks.ceph.cluster1.osd.0.trial042.stderr:2026-02-22T20:22:26.214+0000 7f65e934d640 -1 reset not still connected to 0x55f6fb4441a0 2026-02-22T20:22:27.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:27.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:22:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:27.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:27.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:27.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:27.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2026-02-22T20:22:27.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:27.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:27.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2026-02-22T20:22:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:22:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:22:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:22:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:27.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:27.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:27.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:27.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:27.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:27.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:27.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:22:27.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:22:27.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:27.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:27.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:27.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:27.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2026-02-22T20:22:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:27.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:27.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:22:27.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:27.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:27.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:27.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:27.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2026-02-22T20:22:27.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:27.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:22:27.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:22:27.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:27.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:27.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:27.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:27.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2026-02-22T20:22:27.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:22:27.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:22:27.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:22:27.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:22:27.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_38 2026-02-22T20:22:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:22:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:22:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:22:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:22:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:27.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:22:27.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_38 2026-02-22T20:22:27.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:27.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_38 ']' 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_38' 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:27.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:27.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:27.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:27.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:27.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2026-02-22T20:22:27.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:27.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:27.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:27.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:27.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:27.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:27.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:27.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:27.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:27.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:27.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2026-02-22T20:22:27.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:22:27.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:27.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:27.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:27.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:22:27.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:27.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:22:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:27.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:27.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:27.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:27.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:27.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:27.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:27.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:28.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:28.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:28.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:28.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:28.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_38 2026-02-22T20:22:28.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:22:28.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:28.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:28.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:28.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:22:28.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:28.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:28.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:28.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:28.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_38 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '' 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:28.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:28.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:28.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:28.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:28.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:28.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:28.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:28.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2026-02-22T20:22:28.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:28.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:28.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2026-02-22T20:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:28.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:28.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:28.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:28.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:28.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:28.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=10226' '!=' '' 2026-02-22T20:22:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:22:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=10226' = '' 2026-02-22T20:22:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:28.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:28.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:22:28.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:28.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:22:28.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:28.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:22:28.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:28.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:22:28.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:28.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:22:28.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:28.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:22:28.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:28.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:22:28.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:28.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=10226 2026-02-22T20:22:28.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 10226 2026-02-22T20:22:28.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:22:28.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:28.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_38 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_38 ']' 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_38' 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:28.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:28.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:28.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:28.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:28.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:28.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:22:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:28.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:28.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:28.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:28.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:28.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:28.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:28.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_38 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:28.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:28.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:28.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:28.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:28.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:28.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:28.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_38 2026-02-22T20:22:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '' 2026-02-22T20:22:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:28.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:28.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:28.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:28.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:28.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:28.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:28.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:28.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:28.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:28.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:28.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:28.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:28.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:28.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:28.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:28.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:28.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:28.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:28.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=10226' '!=' 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:28.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:28.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:22:29.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_38 2026-02-22T20:22:29.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:29.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:29.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:29.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_38 ']' 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_38' 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2026-02-22T20:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:29.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:29.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:29.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:29.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:29.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:29.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:29.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:29.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:29.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:29.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:29.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:29.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:29.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2026-02-22T20:22:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:22:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:22:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:29.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:29.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:29.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:29.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:29.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:29.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:29.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:29.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:29.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_38 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:29.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_38 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '' 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:29.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:29.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2026-02-22T20:22:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2026-02-22T20:22:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:29.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:29.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:29.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:29.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:29.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:29.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:29.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:29.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:29.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=10226' '!=' 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:29.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:29.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:22:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_38 2026-02-22T20:22:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_38 ']' 2026-02-22T20:22:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_38' 2026-02-22T20:22:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2026-02-22T20:22:30.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:30.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:30.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:30.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:30.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:30.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:30.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2026-02-22T20:22:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:30.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:30.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:30.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:30.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:30.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:30.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:30.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:30.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:30.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:30.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:30.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2026-02-22T20:22:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:22:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:30.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:30.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:30.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:30.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:30.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:30.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:30.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:30.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:30.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:30.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2026-02-22T20:22:30.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:22:30.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:30.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:30.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:30.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:30.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:30.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:30.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:30.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:30.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:30.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:30.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_38 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:30.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:30.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:30.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:30.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_38 2026-02-22T20:22:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '' 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:30.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:30.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:30.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:30.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:30.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:30.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:30.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:30.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:30.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:30.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:30.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2026-02-22T20:22:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2026-02-22T20:22:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:30.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:30.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:30.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:30.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:30.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:30.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:30.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:30.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:30.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:30.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:30.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:30.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=10226' '!=' 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:22:32.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_38 2026-02-22T20:22:32.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:32.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:32.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:32.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_38 ']' 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_38' 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:32.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:32.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:32.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:32.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:32.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:32.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:32.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:32.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:32.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:32.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:32.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2026-02-22T20:22:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:22:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:22:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:22:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:22:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:32.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:32.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:32.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:32.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:32.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:32.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:32.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:32.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:32.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_38 2026-02-22T20:22:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:22:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:32.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:32.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:32.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:32.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:32.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_38 2026-02-22T20:22:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '' 2026-02-22T20:22:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:32.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:32.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:32.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:32.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:32.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:32.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:32.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:32.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:32.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:32.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2026-02-22T20:22:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2026-02-22T20:22:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:32.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:32.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:32.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:32.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:32.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:32.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:32.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:32.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:32.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:32.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:32.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:32.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:32.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=10226' '!=' 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:22:34.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_38 2026-02-22T20:22:34.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:34.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:34.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:34.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:34.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_38 ']' 2026-02-22T20:22:34.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_38' 2026-02-22T20:22:34.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2026-02-22T20:22:34.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:34.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:34.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:34.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:34.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:34.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:34.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:34.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:34.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:34.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:34.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2026-02-22T20:22:34.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:34.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:34.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:34.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:34.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:34.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:34.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:34.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:34.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:34.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:22:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:34.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:34.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:34.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:34.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:34.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:34.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:34.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:34.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:34.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:34.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:34.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:34.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:34.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:34.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:34.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:34.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:34.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_38 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:34.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:34.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:34.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:34.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:34.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:34.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:34.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_38 2026-02-22T20:22:34.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:34.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:34.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:34.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '' 2026-02-22T20:22:34.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:34.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:34.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:34.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:34.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:34.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:34.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:34.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:34.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:34.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:35.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:35.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:35.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:35.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:35.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2026-02-22T20:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2026-02-22T20:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:35.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:35.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:35.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:35.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:35.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:35.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:35.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:35.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:35.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:35.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:35.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:35.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=10226' '!=' 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:35.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:35.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:22:37.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_38 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_38 ']' 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_38' 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:37.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:37.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:37.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:37.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:37.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:37.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:37.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:37.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:37.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:37.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:37.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:37.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:37.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:37.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:22:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:37.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:37.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:37.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:37.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:37.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:37.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:37.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:37.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:37.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:37.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:37.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:37.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:37.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:37.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_38 2026-02-22T20:22:37.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:22:37.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:37.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:37.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:37.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:22:37.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:37.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:37.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:37.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:22:37.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:37.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:37.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:37.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:37.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:37.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:37.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:37.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:37.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:37.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:37.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:37.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:37.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:37.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:37.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:37.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_38 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '' 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:37.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:37.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:37.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:37.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:37.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2026-02-22T20:22:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2026-02-22T20:22:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:37.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:37.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:37.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:37.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:37.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:37.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:37.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:37.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:37.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:37.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:37.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=10226], [object_number=9, tag_tid=1, entry_tid=10225], [object_number=8, tag_tid=1, entry_tid=10224], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:37.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:37.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:37.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:37.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=10226' '!=' 'object_number=10, tag_tid=1, entry_tid=10226' 2026-02-22T20:22:37.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:37.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 4 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_38 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_38 ']' 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_38' 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2026-02-22T20:22:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:41.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:41.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:41.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:41.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:41.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:41.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:41.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:41.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:41.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:41.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:41.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:41.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:41.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:41.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:22:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:22:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:41.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:41.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:41.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2026-02-22T20:22:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:22:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:22:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:22:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:22:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:22:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:41.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:41.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:22:41.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:41.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2026-02-22T20:22:41.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:22:41.658 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_38 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:22:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:22:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:22:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_38 2026-02-22T20:22:41.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_38 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '' 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:41.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:41.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:41.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:41.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:41.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:41.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:41.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:41.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2026-02-22T20:22:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2026-02-22T20:22:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:22:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_38.status 2026-02-22T20:22:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_38.status 2026-02-22T20:22:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2026-02-22T20:22:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:41.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:41.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:41.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:41.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:41.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10226], [object_number=11, tag_tid=1, entry_tid=10223]]], state=connected] 2026-02-22T20:22:41.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.status 2026-02-22T20:22:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2026-02-22T20:22:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2026-02-22T20:22:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:22:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:22:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:22:42.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2026-02-22T20:22:42.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_38': 2026-02-22T20:22:42.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1208b00db114 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1208b00db114 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:12:39 2026 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:12:39 2026 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:12:39 2026 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1208b00db114 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: dc0a8299-2c38-4a4c-b7a7-39f36fabcc41 2026-02-22T20:22:42.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:22:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:42.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_38 snap 2026-02-22T20:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.export 2026-02-22T20:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_38.export 2026-02-22T20:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_38.export 2026-02-22T20:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.export 2026-02-22T20:22:42.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:22:43.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_38.export 2026-02-22T20:22:43.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:22:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_38.export 2026-02-22T20:22:43.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_38.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_38.export 2026-02-22T20:22:43.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:43.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:22:43.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_39 2026-02-22T20:22:43.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_39 snap 2026-02-22T20:22:43.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:43.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:43.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:43.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:43.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:22:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2026-02-22T20:22:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2026-02-22T20:22:44.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:44.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:44.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:44.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:22:44.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:22:44.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:44.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:22:45.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2026-02-22T20:22:45.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:45.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:45.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:45.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:22:45.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:22:45.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:22:45.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2026-02-22T20:22:45.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:22:45.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:22:45.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:22:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:22:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:22:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:22:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:45.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:45.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:45.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:45.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:45.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:22:45.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:22:45.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:45.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:45.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:45.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:45.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2026-02-22T20:22:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:22:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:22:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:22:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_39 2026-02-22T20:22:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:22:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:22:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:22:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:22:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:45.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:22:46.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_39 2026-02-22T20:22:46.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:46.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:46.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:46.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:46.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_39 ']' 2026-02-22T20:22:46.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_39' 2026-02-22T20:22:46.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2026-02-22T20:22:46.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:46.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:46.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:46.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:46.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:46.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:46.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:46.179 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:46.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:46.183 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:46.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2026-02-22T20:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_39 2026-02-22T20:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '' 2026-02-22T20:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:46.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:46.295 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2026-02-22T20:22:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2026-02-22T20:22:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:46.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28765' '!=' '' 2026-02-22T20:22:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:22:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28765' = '' 2026-02-22T20:22:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:22:46.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:22:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:22:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:46.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:22:46.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:46.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:22:46.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:46.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:22:46.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:46.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:22:46.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:46.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=28765 2026-02-22T20:22:46.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 28765 2026-02-22T20:22:46.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:22:46.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:46.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:22:46.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_39 2026-02-22T20:22:46.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:46.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_39 ']' 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_39' 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:46.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:46.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:46.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:46.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2026-02-22T20:22:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_39 2026-02-22T20:22:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '' 2026-02-22T20:22:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:46.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:46.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:46.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:46.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:46.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:46.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:46.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:46.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:46.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:46.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2026-02-22T20:22:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2026-02-22T20:22:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:46.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:46.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:46.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:46.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:46.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:46.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:46.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:46.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:46.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:46.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:46.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28765' '!=' 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_39 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_39 ']' 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_39' 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:47.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:47.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:47.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:47.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:47.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2026-02-22T20:22:47.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_39 2026-02-22T20:22:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '' 2026-02-22T20:22:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:47.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:47.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:47.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:47.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:47.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:47.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:47.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:47.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:47.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:47.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:47.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:47.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:47.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:47.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2026-02-22T20:22:47.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:47.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:47.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:47.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2026-02-22T20:22:47.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:47.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:47.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:47.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:47.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:47.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:47.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:47.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:47.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:47.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:47.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:47.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:47.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28765' '!=' 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:47.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:47.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_39 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_39 ']' 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_39' 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:48.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:48.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:48.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:48.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:48.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:48.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:48.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:48.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:48.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:48.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:48.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:48.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:48.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2026-02-22T20:22:48.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_39 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '' 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:48.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:48.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:48.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:48.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:48.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:48.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:48.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:48.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:48.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:48.202 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:48.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:48.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2026-02-22T20:22:48.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:48.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:48.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:48.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2026-02-22T20:22:48.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:48.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:48.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:48.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:48.205 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:48.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:48.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:48.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:48.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:48.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:48.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:48.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:48.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:48.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:48.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:48.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28765' '!=' 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:48.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:22:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_39 2026-02-22T20:22:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:49.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_39 ']' 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_39' 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:49.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2026-02-22T20:22:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:49.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:49.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:49.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:49.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:49.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2026-02-22T20:22:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_39 2026-02-22T20:22:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '' 2026-02-22T20:22:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:49.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:49.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2026-02-22T20:22:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2026-02-22T20:22:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:49.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:49.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:49.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:49.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:49.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:49.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=1, entry_tid=28765], [object_number=24, tag_tid=1, entry_tid=28764], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:49.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:49.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:49.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:49.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=25, tag_tid=1, entry_tid=28765' '!=' 'object_number=25, tag_tid=1, entry_tid=28765' 2026-02-22T20:22:49.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:49.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:22:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_39 2026-02-22T20:22:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_39 ']' 2026-02-22T20:22:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_39' 2026-02-22T20:22:51.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2026-02-22T20:22:51.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:51.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:51.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:51.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:51.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:51.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:51.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2026-02-22T20:22:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2026-02-22T20:22:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_39 2026-02-22T20:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '' 2026-02-22T20:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:52.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2026-02-22T20:22:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2026-02-22T20:22:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:22:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_39.status 2026-02-22T20:22:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_39.status 2026-02-22T20:22:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:52.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:52.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2026-02-22T20:22:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:22:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:22:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:52.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:52.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=27, tag_tid=1, entry_tid=28763], [object_number=26, tag_tid=1, entry_tid=28762]]], state=connected] 2026-02-22T20:22:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.status 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:22:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:22:53.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2026-02-22T20:22:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_39': 2026-02-22T20:22:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 12088e8157c0 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.12088e8157c0 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:12:44 2026 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:12:44 2026 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:13:45 2026 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 12088e8157c0 2026-02-22T20:22:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:22:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:22:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 8be94a6c-9210-4009-8f4d-17d3636d093d 2026-02-22T20:22:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:22:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_39 snap 2026-02-22T20:22:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:22:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:22:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.export 2026-02-22T20:22:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_39.export 2026-02-22T20:22:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_39.export 2026-02-22T20:22:53.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.export 2026-02-22T20:22:53.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:22:53.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_39.export 2026-02-22T20:22:53.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:22:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_39.export 2026-02-22T20:22:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_39.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_39.export 2026-02-22T20:22:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_40 2026-02-22T20:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_40 snap 2026-02-22T20:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:22:53.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:22:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2026-02-22T20:22:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:22:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:22:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:22:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2026-02-22T20:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:22:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:22:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2026-02-22T20:22:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:22:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:22:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:22:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:22:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2026-02-22T20:22:54.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:22:54.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:22:54.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:22:54.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:22:54.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:22:54.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_40 2026-02-22T20:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:22:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:22:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:22:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:22:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:22:55.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_40 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_40 ']' 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_40' 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:55.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:55.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2026-02-22T20:22:55.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:55.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:55.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:55.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:55.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:55.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:55.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:55.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:55.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_40 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_40 '' 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_40.status 2026-02-22T20:22:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_40.status 2026-02-22T20:22:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:55.152 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2026-02-22T20:22:55.153 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:55.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:55.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:55.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422]]], state=connected] 2026-02-22T20:22:55.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12424], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422], [object_number=9, tag_tid=1, entry_tid=12421]]], state=connected] 2026-02-22T20:22:55.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:22:55.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2026-02-22T20:22:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2026-02-22T20:22:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_40.status 2026-02-22T20:22:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_40.status 2026-02-22T20:22:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2026-02-22T20:22:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422]]], state=connected] 2026-02-22T20:22:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12424], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422], [object_number=9, tag_tid=1, entry_tid=12421]]], state=connected] 2026-02-22T20:22:55.224 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12424' 2026-02-22T20:22:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12424' 2026-02-22T20:22:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12424' '!=' '' 2026-02-22T20:22:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:22:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12424' = '' 2026-02-22T20:22:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=12424' 2026-02-22T20:22:55.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:22:55.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:22:55.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=3 2026-02-22T20:22:55.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=12424' 2026-02-22T20:22:55.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:22:55.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:55.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:22:55.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:22:55.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:22:55.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:55.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:22:55.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=12424' 2026-02-22T20:22:55.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:22:55.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:22:55.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=12424 2026-02-22T20:22:55.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 3 = 1 -a 1 -le 12424 2026-02-22T20:22:55.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:22:55.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:55.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:22:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_40 2026-02-22T20:22:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_40 ']' 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_40' 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:55.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2026-02-22T20:22:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:55.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:55.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:55.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:55.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:55.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:55.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:55.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_40 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_40 '' 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:55.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:55.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_40.status 2026-02-22T20:22:55.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_40.status 2026-02-22T20:22:55.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:55.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2026-02-22T20:22:55.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:55.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:55.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:55.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422]]], state=connected] 2026-02-22T20:22:55.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12424], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422], [object_number=9, tag_tid=1, entry_tid=12421]]], state=connected] 2026-02-22T20:22:55.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.745 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_40.status 2026-02-22T20:22:55.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_40.status 2026-02-22T20:22:55.747 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:55.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2026-02-22T20:22:55.749 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422]]], state=connected] 2026-02-22T20:22:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12424], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422], [object_number=9, tag_tid=1, entry_tid=12421]]], state=connected] 2026-02-22T20:22:55.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:55.789 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12424' 2026-02-22T20:22:55.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12424' 2026-02-22T20:22:55.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12424' '!=' 'object_number=8, tag_tid=1, entry_tid=12424' 2026-02-22T20:22:55.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:22:55.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_40 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_40 ']' 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_40' 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:22:56.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:22:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:22:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2026-02-22T20:22:56.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:56.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_40 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_40 '' 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_40.status 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_40.status 2026-02-22T20:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:56.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:56.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2026-02-22T20:22:56.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:56.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:56.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:56.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:56.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422]]], state=connected] 2026-02-22T20:22:56.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422]]], state=connected] 2026-02-22T20:22:56.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:56.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:22:56.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_40.status 2026-02-22T20:22:56.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_40.status 2026-02-22T20:22:56.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:22:56.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:56.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2026-02-22T20:22:56.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:56.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 2 2026-02-22T20:22:56.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 2 2026-02-22T20:22:56.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:22:56.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422]]], state=connected] 2026-02-22T20:22:56.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12423], [object_number=10, tag_tid=1, entry_tid=12422]]], state=connected] 2026-02-22T20:22:56.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.status 2026-02-22T20:22:56.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:22:56.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2026-02-22T20:22:56.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:56.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2026-02-22T20:22:56.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:56.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:56.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:56.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:56.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:22:56.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:22:56.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:22:57.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2026-02-22T20:22:57.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_40': 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1208d91f2a1d 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1208d91f2a1d 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:12:54 2026 2026-02-22T20:22:57.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:12:54 2026 2026-02-22T20:22:57.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:22:27 2026 2026-02-22T20:22:57.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:22:57.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1208d91f2a1d 2026-02-22T20:22:57.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:22:57.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:22:57.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: e38c6df2-3875-4498-87c7-1f6794ca6554 2026-02-22T20:22:57.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:22:57.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:57.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_40 snap 2026-02-22T20:22:57.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:57.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:22:57.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:57.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:22:57.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.export 2026-02-22T20:22:57.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_40.export 2026-02-22T20:22:57.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_40.export 2026-02-22T20:22:57.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.export 2026-02-22T20:22:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:22:57.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_40.export 2026-02-22T20:22:57.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:22:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_40.export 2026-02-22T20:22:57.800 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_40.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_40.export 2026-02-22T20:22:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:22:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:22:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_41 2026-02-22T20:22:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_41 snap 2026-02-22T20:22:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:22:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:22:57.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:22:57.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:22:59.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2026-02-22T20:22:59.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:22:59.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:23:00.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2026-02-22T20:23:00.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:00.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:00.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:23:00.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:00.807 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:23:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2026-02-22T20:23:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:23:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:23:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:00.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:00.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:00.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:00.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2026-02-22T20:23:00.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:00.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:23:00.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:00.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:00.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2026-02-22T20:23:00.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:23:00.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:00.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:00.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:00.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:00.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:00.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:00.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:00.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:00.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:00.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:00.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:23:00.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:23:00.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:00.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:00.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:00.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:00.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2026-02-22T20:23:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:23:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2026-02-22T20:23:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:23:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:23:00.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:23:00.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:00.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:00.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:00.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:00.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2026-02-22T20:23:01.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:23:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:01.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:01.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:01.030 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:23:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:23:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:23:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:23:01.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2026-02-22T20:23:01.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:23:01.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:23:01.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:23:01.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:23:01.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:23:01.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_41 2026-02-22T20:23:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:01.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:01.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:01.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:23:01.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:23:01.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:23:01.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:01.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:01.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:01.304 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_41 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_41 ']' 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_41' 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:01.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:01.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2026-02-22T20:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:01.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:01.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:01.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:01.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:01.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:01.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:01.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:01.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:01.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:01.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:01.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_41 2026-02-22T20:23:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_41 2026-02-22T20:23:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:23:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:23:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:01.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:01.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:01.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:01.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:01.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:01.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_41 2026-02-22T20:23:01.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:23:01.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:01.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:01.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:23:01.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:23:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:23:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:01.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:01.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_41 2026-02-22T20:23:01.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:01.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_41 '' 2026-02-22T20:23:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_41.status 2026-02-22T20:23:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_41.status 2026-02-22T20:23:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:01.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:01.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2026-02-22T20:23:01.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:01.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:01.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:01.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:01.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:01.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=1, entry_tid=39748], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746], [object_number=37, tag_tid=1, entry_tid=39745]]], state=connected] 2026-02-22T20:23:01.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_41.status 2026-02-22T20:23:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_41.status 2026-02-22T20:23:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:01.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2026-02-22T20:23:01.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:01.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:01.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:01.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:01.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:01.678 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=1, entry_tid=39748], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746], [object_number=37, tag_tid=1, entry_tid=39745]]], state=connected] 2026-02-22T20:23:01.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:01.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:01.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:01.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=1, entry_tid=39748' '!=' '' 2026-02-22T20:23:01.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:23:01.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=1, entry_tid=39748' = '' 2026-02-22T20:23:01.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:01.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:01.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:01.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:01.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:23:01.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:01.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:01.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:01.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:23:01.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:01.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:01.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:01.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:23:01.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:01.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:01.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:01.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=39748 2026-02-22T20:23:01.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 39748 2026-02-22T20:23:01.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:01.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:01.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:01.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_41 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_41 ']' 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_41' 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:01.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:01.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:01.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:01.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:01.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_41 2026-02-22T20:23:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:01.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:01.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:01.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:02.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:02.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:02.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:02.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:02.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_41 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:02.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:02.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:02.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:02.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:02.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:02.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:02.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:02.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:02.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_41 2026-02-22T20:23:02.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:23:02.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:02.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:02.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:23:02.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:23:02.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:23:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:23:02.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:02.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:02.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:02.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_41 2026-02-22T20:23:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_41 '' 2026-02-22T20:23:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:02.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:02.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_41.status 2026-02-22T20:23:02.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_41.status 2026-02-22T20:23:02.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2026-02-22T20:23:02.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=1, entry_tid=39748], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746], [object_number=37, tag_tid=1, entry_tid=39745]]], state=connected] 2026-02-22T20:23:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:02.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:02.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2026-02-22T20:23:02.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:02.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2026-02-22T20:23:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:02.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_41.status 2026-02-22T20:23:02.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_41.status 2026-02-22T20:23:02.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:02.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:02.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2026-02-22T20:23:02.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=1, entry_tid=39748], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746], [object_number=37, tag_tid=1, entry_tid=39745]]], state=connected] 2026-02-22T20:23:02.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=1, entry_tid=39748' '!=' 'object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:02.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:23:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_41 2026-02-22T20:23:02.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_41 ']' 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_41' 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:02.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:02.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:02.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:02.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:02.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:02.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:02.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:02.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:02.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_41 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:03.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_41 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:03.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:03.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:03.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:03.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:03.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:03.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:03.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_41 2026-02-22T20:23:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:23:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:23:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:23:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:23:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:23:03.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:03.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:03.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:03.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:03.173 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_41 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_41 '' 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:03.174 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_41.status 2026-02-22T20:23:03.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_41.status 2026-02-22T20:23:03.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:03.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:03.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2026-02-22T20:23:03.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=1, entry_tid=39748], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746], [object_number=37, tag_tid=1, entry_tid=39745]]], state=connected] 2026-02-22T20:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:03.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:03.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2026-02-22T20:23:03.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:03.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:03.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:03.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2026-02-22T20:23:03.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:03.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:03.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:03.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:03.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_41.status 2026-02-22T20:23:03.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_41.status 2026-02-22T20:23:03.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:03.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:03.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2026-02-22T20:23:03.212 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:03.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=36, tag_tid=1, entry_tid=39748], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746], [object_number=37, tag_tid=1, entry_tid=39745]]], state=connected] 2026-02-22T20:23:03.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:03.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:03.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:03.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=36, tag_tid=1, entry_tid=39748' '!=' 'object_number=36, tag_tid=1, entry_tid=39748' 2026-02-22T20:23:03.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:03.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_41 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_41 ']' 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_41' 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:04.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:04.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:04.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:04.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:04.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:04.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:04.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:04.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:04.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_41 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:04.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:04.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:04.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:04.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:04.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:04.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:04.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_41 2026-02-22T20:23:04.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:23:04.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:04.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:23:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:23:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:04.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:04.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:04.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:23:04.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:04.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:04.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:23:04.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:04.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:04.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:04.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:04.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:04.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:04.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:04.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:04.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:04.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:04.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:04.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:04.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:04.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:04.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2026-02-22T20:23:04.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:04.255 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:04.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_41 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:23:04.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:23:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:23:04.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:04.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:04.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:23:04.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_41 2026-02-22T20:23:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_41 2026-02-22T20:23:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_41 '' 2026-02-22T20:23:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:04.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:04.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:04.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_41.status 2026-02-22T20:23:04.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_41.status 2026-02-22T20:23:04.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:04.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:04.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2026-02-22T20:23:04.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:04.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:04.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:04.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:04.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:04.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:04.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:04.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_41.status 2026-02-22T20:23:04.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_41.status 2026-02-22T20:23:04.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:04.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:04.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2026-02-22T20:23:04.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:04.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:04.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:04.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:04.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:04.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=39, tag_tid=1, entry_tid=39747], [object_number=38, tag_tid=1, entry_tid=39746]]], state=connected] 2026-02-22T20:23:04.399 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.status 2026-02-22T20:23:04.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:23:04.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:23:05.403 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2026-02-22T20:23:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_41': 2026-02-22T20:23:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:23:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:23:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aacafc5bac 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aacafc5bac 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:12:59 2026 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:12:59 2026 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:14:00 2026 2026-02-22T20:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:23:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aacafc5bac 2026-02-22T20:23:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:23:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:23:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: f5a5af14-b451-4c74-925c-d28d0a97fd2d 2026-02-22T20:23:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_41 snap 2026-02-22T20:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.export 2026-02-22T20:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_41.export 2026-02-22T20:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_41.export 2026-02-22T20:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.export 2026-02-22T20:23:05.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:23:05.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_41.export 2026-02-22T20:23:05.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:23:05.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_41.export 2026-02-22T20:23:05.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_41.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_41.export 2026-02-22T20:23:05.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:05.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:23:05.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_42 2026-02-22T20:23:05.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_42 snap 2026-02-22T20:23:05.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:05.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:05.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:05.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:05.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:23:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:23:09.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2026-02-22T20:23:09.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:09.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:09.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:09.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:23:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:23:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:23:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2026-02-22T20:23:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:23:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:23:09.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:09.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:09.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:23:09.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:23:09.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:09.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:23:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:23:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2026-02-22T20:23:09.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:23:09.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:23:09.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:23:09.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:23:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:23:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:23:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:23:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2026-02-22T20:23:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_42 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_42 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_42 ']' 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_42' 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:10.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:10.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:10.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:10.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:10.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2026-02-22T20:23:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_42 2026-02-22T20:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '' 2026-02-22T20:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:10.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:10.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:10.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:10.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:10.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:10.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:10.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:10.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:10.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2026-02-22T20:23:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2026-02-22T20:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:10.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:10.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:10.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:10.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:10.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:10.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:10.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:10.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:10.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=18, tag_tid=1, entry_tid=19578' '!=' '' 2026-02-22T20:23:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:23:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=18, tag_tid=1, entry_tid=19578' = '' 2026-02-22T20:23:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:10.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:23:10.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:10.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:10.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:10.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:23:10.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:10.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:10.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:10.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:23:10.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:10.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:10.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:10.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=19578 2026-02-22T20:23:10.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 19578 2026-02-22T20:23:10.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:10.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:10.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_42 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_42 ']' 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_42' 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2026-02-22T20:23:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2026-02-22T20:23:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_42 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '' 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:10.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:10.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:10.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:10.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:10.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:10.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:10.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:10.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:10.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:10.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:10.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:10.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:10.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=18, tag_tid=1, entry_tid=19578' '!=' 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:10.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:10.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_42 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_42 ']' 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_42' 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:11.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:11.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:11.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:11.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:11.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:11.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:11.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:11.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2026-02-22T20:23:11.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:11.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_42 2026-02-22T20:23:11.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '' 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:11.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:11.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:11.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:11.122 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:11.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:11.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:11.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:11.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:11.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:11.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:11.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:11.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:11.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:11.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:11.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:11.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:11.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:11.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:11.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=18, tag_tid=1, entry_tid=19578' '!=' 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:11.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:11.199 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:23:11.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8641467756343615) > weight (0.25) 2026-02-22T20:23:11.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6938523460692474) > weight (0.25) 2026-02-22T20:23:11.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8100556896461109) > weight (0.25) 2026-02-22T20:23:11.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9212028146506447) > weight (0.25) 2026-02-22T20:23:11.393 INFO:tasks.rbd_mirror_thrash:waiting for 75.444257553624 secs before thrashing 2026-02-22T20:23:12.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_42 2026-02-22T20:23:12.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:12.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:12.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:12.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:12.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_42 ']' 2026-02-22T20:23:12.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_42' 2026-02-22T20:23:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2026-02-22T20:23:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:12.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:12.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:12.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:12.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:12.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:12.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:12.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:12.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:12.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2026-02-22T20:23:12.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:12.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_42 2026-02-22T20:23:12.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:12.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:12.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:12.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '' 2026-02-22T20:23:12.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:12.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:12.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:12.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:12.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:12.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:12.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:12.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:12.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:12.094 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:12.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2026-02-22T20:23:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2026-02-22T20:23:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:12.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:12.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:12.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:12.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:12.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:12.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:12.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:12.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:12.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:12.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:12.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:12.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=18, tag_tid=1, entry_tid=19578' '!=' 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:12.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:12.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:23:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_42 2026-02-22T20:23:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_42 ']' 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_42' 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2026-02-22T20:23:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:13.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:13.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:13.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:13.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2026-02-22T20:23:13.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_42 2026-02-22T20:23:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '' 2026-02-22T20:23:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:13.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:13.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:13.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:13.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:13.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:13.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:13.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:13.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:13.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:13.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2026-02-22T20:23:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2026-02-22T20:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:13.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:13.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:13.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:13.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:13.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:13.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=18, tag_tid=1, entry_tid=19578' '!=' 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:13.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:13.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_42 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_42 ']' 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_42' 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2026-02-22T20:23:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:15.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:15.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:15.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:15.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:15.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:15.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:15.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:15.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2026-02-22T20:23:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_42 2026-02-22T20:23:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '' 2026-02-22T20:23:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:16.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:16.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:16.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:16.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:16.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:16.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:16.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:16.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:16.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2026-02-22T20:23:16.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:16.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:16.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:16.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2026-02-22T20:23:16.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:16.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:16.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:16.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:16.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:16.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:16.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:16.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:16.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:16.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:16.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:16.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:16.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:16.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:16.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:16.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:16.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:16.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:16.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=18, tag_tid=1, entry_tid=19578' '!=' 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:16.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:16.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:23:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_42 2026-02-22T20:23:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_42 ']' 2026-02-22T20:23:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_42' 2026-02-22T20:23:18.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2026-02-22T20:23:18.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:18.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:18.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:18.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:18.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:18.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:18.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:18.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:18.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:18.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:18.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:18.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:18.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2026-02-22T20:23:18.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:18.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_42 2026-02-22T20:23:18.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:18.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:18.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:18.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '' 2026-02-22T20:23:18.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:18.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:18.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:18.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:18.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:18.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:18.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:18.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:18.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:18.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:18.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:18.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:18.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:18.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:18.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:18.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:18.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2026-02-22T20:23:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2026-02-22T20:23:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:18.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:18.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:18.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:18.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:18.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=18, tag_tid=1, entry_tid=19578], [object_number=17, tag_tid=1, entry_tid=19577], [object_number=16, tag_tid=1, entry_tid=19576], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:18.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=18, tag_tid=1, entry_tid=19578' '!=' 'object_number=18, tag_tid=1, entry_tid=19578' 2026-02-22T20:23:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:18.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 4 2026-02-22T20:23:22.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_42 2026-02-22T20:23:22.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_42 ']' 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_42' 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:22.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:22.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:22.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:22.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:22.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:22.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:22.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:22.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:22.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:22.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:22.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:22.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:22.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:22.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:22.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:22.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:22.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2026-02-22T20:23:22.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_42 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '' 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:22.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:22.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:22.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:22.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:22.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:22.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:22.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:22.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:22.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:22.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2026-02-22T20:23:22.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_42.status 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_42.status 2026-02-22T20:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:22.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:22.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2026-02-22T20:23:22.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:22.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 4 2026-02-22T20:23:22.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 4 2026-02-22T20:23:22.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:22.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:22.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=19578], [object_number=19, tag_tid=1, entry_tid=19575]]], state=connected] 2026-02-22T20:23:22.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.status 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:23:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:23:23.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2026-02-22T20:23:23.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_42': 2026-02-22T20:23:23.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:23:23.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:23:23.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1127fece9672 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1127fece9672 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:13:14 2026 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:13:14 2026 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:13:14 2026 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:23:23.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1127fece9672 2026-02-22T20:23:23.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:23:23.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:23:23.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: b2d818bc-b9ad-4bf7-ae4c-495459298ff9 2026-02-22T20:23:23.421 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:23:23.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_42 snap 2026-02-22T20:23:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:23:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:23:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.export 2026-02-22T20:23:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_42.export 2026-02-22T20:23:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_42.export 2026-02-22T20:23:23.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.export 2026-02-22T20:23:23.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:23:23.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_42.export 2026-02-22T20:23:23.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:23:23.737 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_42.export 2026-02-22T20:23:23.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_42.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_42.export 2026-02-22T20:23:23.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:23.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:23:23.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_43 2026-02-22T20:23:23.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_43 snap 2026-02-22T20:23:23.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:23.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:23.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:23.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:23.841 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:23:27.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2026-02-22T20:23:27.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:27.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:27.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:27.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2026-02-22T20:23:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:23:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:23:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:27.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:23:28.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:28.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:28.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:23:28.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:23:28.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:28.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:23:28.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:28.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:23:28.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2026-02-22T20:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:23:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:23:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:23:28.854 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:28.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:28.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:28.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:28.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2026-02-22T20:23:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:23:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2026-02-22T20:23:28.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:28.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:23:28.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:29.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:29.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:23:29.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:23:29.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:29.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:29.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:29.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:29.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2026-02-22T20:23:29.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:23:29.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:23:29.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:23:29.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:23:29.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_43 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:23:29.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:23:29.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:29.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:29.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_43 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_43 ']' 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_43' 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:29.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:29.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:29.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2026-02-22T20:23:29.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:29.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:29.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:29.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:29.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:29.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:29.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:29.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:29.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:29.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:29.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:29.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:29.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:29.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:29.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:29.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2026-02-22T20:23:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:23:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_43 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '' 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:29.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:29.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:29.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:29.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:29.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978], [object_number=21, tag_tid=1, entry_tid=24977], [object_number=20, tag_tid=1, entry_tid=24976]]], state=connected] 2026-02-22T20:23:29.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:23:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2026-02-22T20:23:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2026-02-22T20:23:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:29.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:29.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:29.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:29.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:29.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978], [object_number=21, tag_tid=1, entry_tid=24977], [object_number=20, tag_tid=1, entry_tid=24976]]], state=connected] 2026-02-22T20:23:29.581 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:29.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:29.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:29.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=23, tag_tid=1, entry_tid=24979' '!=' '' 2026-02-22T20:23:29.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:23:29.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=23, tag_tid=1, entry_tid=24979' = '' 2026-02-22T20:23:29.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:29.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:23:29.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:29.584 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:29.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:23:29.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:29.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:29.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:29.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:23:29.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:23:29.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:29.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:29.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:23:29.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:29.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:29.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=24979 2026-02-22T20:23:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 24979 2026-02-22T20:23:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:29.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_43 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_43 ']' 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_43' 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:29.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:29.808 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:29.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:29.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:29.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2026-02-22T20:23:29.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:29.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:29.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:29.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:29.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:29.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:29.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:29.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:29.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:29.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:29.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:29.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:29.877 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:29.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_43 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '' 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:30.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:30.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:30.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:30.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:30.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:30.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:30.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978], [object_number=21, tag_tid=1, entry_tid=24977], [object_number=20, tag_tid=1, entry_tid=24976]]], state=connected] 2026-02-22T20:23:30.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.055 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:23:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2026-02-22T20:23:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2026-02-22T20:23:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:30.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:30.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:30.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:30.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978], [object_number=21, tag_tid=1, entry_tid=24977], [object_number=20, tag_tid=1, entry_tid=24976]]], state=connected] 2026-02-22T20:23:30.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:30.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:30.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=23, tag_tid=1, entry_tid=24979' '!=' 'object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:30.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:30.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:23:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_43 2026-02-22T20:23:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_43 ']' 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_43' 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2026-02-22T20:23:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:30.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:30.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:30.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:30.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:30.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:30.568 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:30.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:30.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:30.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:30.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:30.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:30.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:30.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:30.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:30.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:30.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:30.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:30.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:30.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:30.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:30.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:30.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:30.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:30.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:30.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:30.657 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:30.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:30.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_43 2026-02-22T20:23:30.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:30.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:30.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:30.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '' 2026-02-22T20:23:30.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:30.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:30.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:30.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:30.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:30.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:30.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:30.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:30.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:30.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:30.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:30.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:30.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978], [object_number=21, tag_tid=1, entry_tid=24977], [object_number=20, tag_tid=1, entry_tid=24976]]], state=connected] 2026-02-22T20:23:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:30.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:30.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:30.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:30.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:30.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:30.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978], [object_number=21, tag_tid=1, entry_tid=24977], [object_number=20, tag_tid=1, entry_tid=24976]]], state=connected] 2026-02-22T20:23:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=23, tag_tid=1, entry_tid=24979' '!=' 'object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:23:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_43 2026-02-22T20:23:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_43 ']' 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_43' 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:31.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:31.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:31.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:31.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:31.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:31.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:31.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:31.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2026-02-22T20:23:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:31.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:31.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:31.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:31.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:31.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:31.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:31.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:31.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2026-02-22T20:23:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:23:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:23:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:31.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:31.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:31.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:31.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:31.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:31.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_43 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '' 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:31.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:31.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:31.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:31.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:31.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:31.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:31.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:31.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:31.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:31.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978], [object_number=21, tag_tid=1, entry_tid=24977], [object_number=20, tag_tid=1, entry_tid=24976]]], state=connected] 2026-02-22T20:23:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:23:31.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2026-02-22T20:23:31.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:31.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:31.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:31.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2026-02-22T20:23:31.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:31.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:31.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:31.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:31.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:31.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:31.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:31.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:31.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:31.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:31.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:31.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:31.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:31.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:31.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978], [object_number=21, tag_tid=1, entry_tid=24977], [object_number=20, tag_tid=1, entry_tid=24976]]], state=connected] 2026-02-22T20:23:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:31.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:31.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:31.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=23, tag_tid=1, entry_tid=24979' '!=' 'object_number=23, tag_tid=1, entry_tid=24979' 2026-02-22T20:23:31.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:31.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:23:33.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_43 2026-02-22T20:23:33.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:33.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:33.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:33.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:33.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_43 ']' 2026-02-22T20:23:33.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_43' 2026-02-22T20:23:33.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2026-02-22T20:23:33.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:33.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:33.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:33.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2026-02-22T20:23:33.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:33.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:33.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:33.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:33.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:33.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:33.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:33.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:33.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:33.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:33.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:33.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:33.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-22T20:23:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:33.656 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:23:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:23:33.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:33.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:33.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:23:33.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2026-02-22T20:23:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_43 2026-02-22T20:23:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '' 2026-02-22T20:23:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:33.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:33.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:33.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:33.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:33.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:33.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:33.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:33.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:33.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:33.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:33.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:33.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_43.status 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_43.status 2026-02-22T20:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:33.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:33.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2026-02-22T20:23:33.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 5 2026-02-22T20:23:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 5 2026-02-22T20:23:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=23, tag_tid=1, entry_tid=24979], [object_number=22, tag_tid=1, entry_tid=24978]]], state=connected] 2026-02-22T20:23:33.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.status 2026-02-22T20:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=1' 2026-02-22T20:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2026-02-22T20:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:33.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:23:33.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:23:33.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:23:34.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2026-02-22T20:23:34.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_43': 2026-02-22T20:23:34.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:23:34.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:23:34.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:23:34.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e3fb56f0c9 2026-02-22T20:23:34.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e3fb56f0c9 2026-02-22T20:23:34.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:23:34.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:23:34.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:13:21 2026 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:13:21 2026 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:14:22 2026 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e3fb56f0c9 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 6163fd51-4946-430f-935c-79ec5ebc0c88 2026-02-22T20:23:34.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:23:34.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:34.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_43 snap 2026-02-22T20:23:34.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:34.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:23:34.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:34.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:23:34.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.export 2026-02-22T20:23:34.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_43.export 2026-02-22T20:23:34.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_43.export 2026-02-22T20:23:34.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.export 2026-02-22T20:23:34.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:23:35.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_43.export 2026-02-22T20:23:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:23:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_43.export 2026-02-22T20:23:35.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_43.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_43.export 2026-02-22T20:23:35.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:23:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_44 2026-02-22T20:23:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_44 snap 2026-02-22T20:23:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:35.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:23:36.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:23:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:23:37.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:37.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:23:37.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:37.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:37.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:37.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:37.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2026-02-22T20:23:37.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-22T20:23:37.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:23:37.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2026-02-22T20:23:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:23:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:23:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:23:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:37.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:37.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2026-02-22T20:23:37.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:23:37.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:23:37.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:23:37.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:23:37.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:23:37.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:23:37.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:23:37.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_44 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:37.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:23:37.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:23:37.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:37.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:37.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:38.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_44 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_44 ']' 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_44' 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:38.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:38.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:38.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:38.198 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:38.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:38.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:38.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-22T20:23:38.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:38.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:38.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:38.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:38.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:38.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:38.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:38.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_44 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '' 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:38.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2026-02-22T20:23:38.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:38.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:38.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:38.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2026-02-22T20:23:38.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:38.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:38.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:38.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:38.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:38.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:38.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:38.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:38.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:38.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:38.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:38.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:38.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:38.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=15396' '!=' '' 2026-02-22T20:23:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:23:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=15396' = '' 2026-02-22T20:23:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:38.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:38.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:38.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:38.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:23:38.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:38.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:38.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:38.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:23:38.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:38.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:38.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:38.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:23:38.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:38.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:38.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:38.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=15396 2026-02-22T20:23:38.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 15396 2026-02-22T20:23:38.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:38.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:38.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_44 2026-02-22T20:23:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_44 ']' 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_44' 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:38.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:38.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:38.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:38.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-22T20:23:38.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:38.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:38.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:38.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:38.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_44 2026-02-22T20:23:38.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:38.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:38.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:38.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '' 2026-02-22T20:23:38.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:38.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:38.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:38.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:38.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:38.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:38.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:38.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:38.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:38.820 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:38.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:38.828 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:38.858 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=15396' '!=' 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:38.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_44 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_44 ']' 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_44' 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:39.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:39.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:39.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:39.264 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:39.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:39.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:39.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:39.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:39.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:39.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:39.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:39.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:39.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-22T20:23:39.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:39.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:39.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:39.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2026-02-22T20:23:39.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:39.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:39.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:39.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:39.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:39.339 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:39.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:39.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:39.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:39.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:39.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:39.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_44 2026-02-22T20:23:39.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:39.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '' 2026-02-22T20:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:39.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:39.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:39.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:39.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:39.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:39.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:39.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:39.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=15396' '!=' 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:39.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:39.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_44 2026-02-22T20:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_44 ']' 2026-02-22T20:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_44' 2026-02-22T20:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2026-02-22T20:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:40.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:40.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:40.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:40.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:40.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:40.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:40.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:40.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:40.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:40.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:40.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:40.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:40.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:40.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:40.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:40.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-22T20:23:40.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:40.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:40.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:40.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:40.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_44 2026-02-22T20:23:40.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '' 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:40.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:40.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:40.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:40.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:40.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:40.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2026-02-22T20:23:40.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:40.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2026-02-22T20:23:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:40.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:40.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:40.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:40.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=15396' '!=' 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_44 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_44 ']' 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_44' 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:42.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-22T20:23:42.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:42.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:42.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:42.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:42.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:42.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:42.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:42.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:42.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:42.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_44 2026-02-22T20:23:42.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '' 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:42.289 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:42.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:42.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:42.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:42.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:42.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:42.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:42.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:42.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:42.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15396], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394], [object_number=13, tag_tid=1, entry_tid=15393]]], state=connected] 2026-02-22T20:23:42.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:42.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:42.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:42.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=15396' '!=' 'object_number=12, tag_tid=1, entry_tid=15396' 2026-02-22T20:23:42.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:42.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_44 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_44 ']' 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_44' 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:44.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:44.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:44.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:44.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-22T20:23:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2026-02-22T20:23:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:44.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:44.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:44.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:44.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:44.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:44.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2026-02-22T20:23:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_44 2026-02-22T20:23:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '' 2026-02-22T20:23:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:44.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:44.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:44.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:44.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:44.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:44.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:44.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:44.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:44.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:44.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:44.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_44.status 2026-02-22T20:23:44.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_44.status 2026-02-22T20:23:44.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2026-02-22T20:23:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 3 2026-02-22T20:23:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 3 2026-02-22T20:23:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15395], [object_number=14, tag_tid=1, entry_tid=15394]]], state=connected] 2026-02-22T20:23:44.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.status 2026-02-22T20:23:44.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:44.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2026-02-22T20:23:44.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:44.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2026-02-22T20:23:44.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:44.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:44.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:44.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:44.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:23:44.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:23:44.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:23:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_44': 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1208452bfa0b 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1208452bfa0b 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:13:28 2026 2026-02-22T20:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:13:28 2026 2026-02-22T20:23:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:13:28 2026 2026-02-22T20:23:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:23:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1208452bfa0b 2026-02-22T20:23:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:23:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:23:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 2431aa39-f0ed-44ae-9277-25b80fc6e91b 2026-02-22T20:23:45.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:23:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_44 snap 2026-02-22T20:23:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:23:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:23:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.export 2026-02-22T20:23:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_44.export 2026-02-22T20:23:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_44.export 2026-02-22T20:23:45.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.export 2026-02-22T20:23:45.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:23:45.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_44.export 2026-02-22T20:23:45.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:23:45.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_44.export 2026-02-22T20:23:46.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_44.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_44.export 2026-02-22T20:23:46.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:46.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:23:46.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_45 2026-02-22T20:23:46.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_45 snap 2026-02-22T20:23:46.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:46.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:46.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:46.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:23:46.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:23:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2026-02-22T20:23:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:23:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:23:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2026-02-22T20:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:23:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:23:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:23:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2026-02-22T20:23:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:23:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:50.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:23:50.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:23:50.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:50.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:50.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:50.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:50.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2026-02-22T20:23:50.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-22T20:23:50.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:23:50.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:50.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:50.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:50.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:50.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:50.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:50.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2026-02-22T20:23:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:23:51.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:23:51.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:23:51.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:23:51.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:23:51.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_45 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:51.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:51.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:23:51.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:23:51.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:51.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:51.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:51.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_45 2026-02-22T20:23:51.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:51.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:51.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_45 ']' 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_45' 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:51.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:51.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:51.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-22T20:23:51.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:51.302 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2026-02-22T20:23:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:51.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:51.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_45 2026-02-22T20:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '' 2026-02-22T20:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:51.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:51.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:51.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:51.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:51.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:51.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:51.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:51.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:51.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2026-02-22T20:23:51.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:51.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:51.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:51.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2026-02-22T20:23:51.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:51.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:51.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:51.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:51.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:51.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:51.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:51.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:51.432 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:51.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:51.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:51.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:51.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:51.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:51.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:51.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41362' '!=' '' 2026-02-22T20:23:51.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:23:51.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41362' = '' 2026-02-22T20:23:51.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:51.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:51.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:51.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:51.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:23:51.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:51.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:23:51.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:51.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:23:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:51.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:23:51.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:51.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:23:51.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:23:51.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41362 2026-02-22T20:23:51.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 41362 2026-02-22T20:23:51.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:23:51.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:51.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_45 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_45 ']' 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_45' 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2026-02-22T20:23:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:51.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:51.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:51.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:51.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:51.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-22T20:23:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:51.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:51.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_45 2026-02-22T20:23:51.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:51.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '' 2026-02-22T20:23:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:51.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:51.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:51.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:51.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:51.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2026-02-22T20:23:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:51.881 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:51.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2026-02-22T20:23:51.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:51.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:51.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:51.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:51.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:51.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:51.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:51.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:51.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:51.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:51.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:51.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:51.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:51.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41362' '!=' 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:23:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_45 2026-02-22T20:23:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_45 ']' 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_45' 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:52.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:52.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:52.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-22T20:23:52.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:52.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2026-02-22T20:23:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:52.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:52.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:52.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_45 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '' 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:52.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:52.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:52.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:52.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:52.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:52.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:52.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:52.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:52.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:52.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:52.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:52.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:52.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:52.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:52.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:52.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:52.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:52.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:52.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:52.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:52.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:52.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:52.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:52.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:52.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:52.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:52.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:52.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41362' '!=' 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:52.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:52.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_45 2026-02-22T20:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_45 ']' 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_45' 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:53.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:53.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-22T20:23:53.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:53.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:53.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:53.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:53.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:53.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:53.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:53.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:53.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_45 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '' 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:53.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:53.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:53.485 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:53.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:53.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:53.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:53.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:53.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:53.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:53.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:53.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:53.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:53.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:53.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:53.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:53.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:53.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:53.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41362' '!=' 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:53.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:53.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_45 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_45 ']' 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_45' 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:55.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:55.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:55.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:55.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:55.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:55.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:55.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:55.379 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:55.389 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:55.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:55.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:55.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:55.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-22T20:23:55.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:55.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:55.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:55.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:55.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:55.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:55.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:55.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:55.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_45 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '' 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:55.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:55.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:55.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:55.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:55.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:55.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:55.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:55.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:55.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:55.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:55.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:55.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:55.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:55.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:55.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:55.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:55.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:55.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:55.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:55.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:55.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:55.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41362' '!=' 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:55.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:55.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:23:55.622 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9574598112479566) > weight (0.25) 2026-02-22T20:23:55.622 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7370983858780907) > weight (0.25) 2026-02-22T20:23:55.622 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2026-02-22T20:23:55.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2026-02-22T20:23:55.622 INFO:tasks.rbd_mirror_thrash:waiting for 0.8174362024644988 secs before reviving daemons 2026-02-22T20:23:55.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:23:55.616+0000 7f4f02d62640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 107478) UID: 1000 2026-02-22T20:23:56.440 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2026-02-22T20:23:56.440 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:23:56.440 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2026-02-22T20:23:56.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2026-02-22T20:23:56.441 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2026-02-22T20:23:56.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2026-02-22T20:23:56.442 INFO:tasks.rbd_mirror_thrash:waiting for 73.15096513965403 secs before thrashing 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_45 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_45 ']' 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_45' 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:57.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:57.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:57.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:57.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:57.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:57.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:57.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:57.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:57.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:57.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:57.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:57.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:57.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:57.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:57.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:57.615 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:57.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:57.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:57.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:57.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-22T20:23:57.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:57.670 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:57.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:57.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2026-02-22T20:23:57.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:57.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:57.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:57.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:57.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:57.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:57.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:57.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:57.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:57.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:57.676 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:57.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:57.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:57.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:57.684 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_45 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '' 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:57.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:57.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:57.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:57.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:57.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:57.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:57.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:57.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:57.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:57.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:57.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:23:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2026-02-22T20:23:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2026-02-22T20:23:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:23:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:23:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:23:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:23:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:23:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:23:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:23:57.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:57.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:23:57.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:57.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:23:57.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:23:57.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:23:57.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:57.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41362], [object_number=37, tag_tid=1, entry_tid=41361], [object_number=36, tag_tid=1, entry_tid=41360], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:23:57.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:23:57.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:57.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:57.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41362' '!=' 'object_number=38, tag_tid=1, entry_tid=41362' 2026-02-22T20:23:57.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:23:57.839 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:23:59.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_45 2026-02-22T20:23:59.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:59.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:23:59.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:23:59.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:23:59.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_45 ']' 2026-02-22T20:23:59.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_45' 2026-02-22T20:23:59.842 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2026-02-22T20:23:59.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:23:59.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:59.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:23:59.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:23:59.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:23:59.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:23:59.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:23:59.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:23:59.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:23:59.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:59.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2026-02-22T20:23:59.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:23:59.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:59.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:59.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:23:59.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:23:59.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:59.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:23:59.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:23:59.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:59.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:59.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:23:59.857 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2026-02-22T20:23:59.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-22T20:23:59.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:23:59.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:23:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:23:59.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:23:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:23:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2026-02-22T20:24:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_45 2026-02-22T20:24:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:24:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '' 2026-02-22T20:24:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:24:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:24:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:24:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:00.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:24:00.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:24:00.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:24:00.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:00.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:00.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:00.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:24:00.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:24:00.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:24:00.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2026-02-22T20:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2026-02-22T20:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_45.status 2026-02-22T20:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_45.status 2026-02-22T20:24:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:24:00.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2026-02-22T20:24:00.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:24:00.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:00.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:00.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:00.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:24:00.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41362], [object_number=39, tag_tid=1, entry_tid=41359]]], state=connected] 2026-02-22T20:24:00.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.status 2026-02-22T20:24:00.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:00.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:00.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:00.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2026-02-22T20:24:00.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:00.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:00.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:24:00.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:00.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:00.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:24:00.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:01.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_45': 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aaf036852b 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aaf036852b 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:13:33 2026 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:13:33 2026 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:14:34 2026 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aaf036852b 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:24:01.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:24:01.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 8dac6671-acb3-414e-886b-992af108772e 2026-02-22T20:24:01.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_45 snap 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.export 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_45.export 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_45.export 2026-02-22T20:24:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.export 2026-02-22T20:24:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:01.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_45.export 2026-02-22T20:24:01.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_45.export 2026-02-22T20:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_45.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_45.export 2026-02-22T20:24:01.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:01.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:01.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_46 2026-02-22T20:24:01.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_46 snap 2026-02-22T20:24:01.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:01.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:01.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:01.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:01.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:24:04.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2026-02-22T20:24:04.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:04.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:04.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:04.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2026-02-22T20:24:04.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:04.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:04.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:04.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:24:04.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:04.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:04.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:05.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2026-02-22T20:24:05.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:05.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:05.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:05.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:24:05.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:24:05.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:24:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2026-02-22T20:24:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:24:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:24:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:24:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:24:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:24:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:24:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2026-02-22T20:24:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2026-02-22T20:24:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:24:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:24:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:24:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:24:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:24:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:05.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:05.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:05.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:05.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2026-02-22T20:24:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:06.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:24:06.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:06.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:06.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:06.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:06.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2026-02-22T20:24:06.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:24:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:24:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:24:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:24:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_46 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:24:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:06.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:24:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_46 2026-02-22T20:24:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_46 ']' 2026-02-22T20:24:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_46' 2026-02-22T20:24:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2026-02-22T20:24:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2026-02-22T20:24:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:06.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:06.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:06.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:06.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:06.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:06.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:06.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:06.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:06.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2026-02-22T20:24:06.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:06.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:06.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:06.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:06.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:06.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:06.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:06.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_46 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_46 '' 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_46.status 2026-02-22T20:24:06.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_46.status 2026-02-22T20:24:06.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:06.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:06.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2026-02-22T20:24:06.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:06.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:24:06.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:24:06.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:06.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:06.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30380], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378], [object_number=29, tag_tid=1, entry_tid=30377]]], state=connected] 2026-02-22T20:24:06.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:06.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:24:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2026-02-22T20:24:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2026-02-22T20:24:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:06.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_46.status 2026-02-22T20:24:06.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_46.status 2026-02-22T20:24:06.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:06.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:06.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2026-02-22T20:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:06.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:24:06.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:24:06.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:06.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:06.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30380], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378], [object_number=29, tag_tid=1, entry_tid=30377]]], state=connected] 2026-02-22T20:24:06.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=28, tag_tid=1, entry_tid=30380' '!=' '' 2026-02-22T20:24:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:24:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=28, tag_tid=1, entry_tid=30380' = '' 2026-02-22T20:24:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:24:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:24:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:06.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:24:06.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:06.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:24:06.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:06.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:24:06.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:24:06.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:24:06.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:06.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:24:06.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:06.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:24:06.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:06.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=30380 2026-02-22T20:24:06.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 30380 2026-02-22T20:24:06.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:24:06.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:06.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_46 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_46 ']' 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_46' 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:06.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:06.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:06.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:06.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:06.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:06.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:06.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:06.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:06.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:06.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:06.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:06.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:06.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:06.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:06.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:06.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:06.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:06.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:07.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:07.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:07.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:07.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_46 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_46 '' 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:07.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:07.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_46.status 2026-02-22T20:24:07.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_46.status 2026-02-22T20:24:07.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:07.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2026-02-22T20:24:07.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:24:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:24:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30380], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378], [object_number=29, tag_tid=1, entry_tid=30377]]], state=connected] 2026-02-22T20:24:07.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_46.status 2026-02-22T20:24:07.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_46.status 2026-02-22T20:24:07.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:07.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2026-02-22T20:24:07.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:24:07.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:24:07.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:07.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:07.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30380], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378], [object_number=29, tag_tid=1, entry_tid=30377]]], state=connected] 2026-02-22T20:24:07.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=28, tag_tid=1, entry_tid=30380' '!=' 'object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:24:07.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_46 2026-02-22T20:24:07.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:07.550 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_46 ']' 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_46' 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:07.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:07.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:07.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:07.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:07.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:07.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:07.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:07.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:07.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:07.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:07.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:07.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:07.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:07.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:07.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:07.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:07.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:07.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:07.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:07.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:07.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:07.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:07.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:07.697 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:07.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:07.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2026-02-22T20:24:07.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:07.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:07.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:07.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:07.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:07.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:07.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:07.700 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:07.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:07.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:07.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:07.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:07.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:07.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_46 2026-02-22T20:24:07.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:07.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_46 '' 2026-02-22T20:24:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_46.status 2026-02-22T20:24:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_46.status 2026-02-22T20:24:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:07.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2026-02-22T20:24:07.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:24:07.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:24:07.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:07.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:07.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30380], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378], [object_number=29, tag_tid=1, entry_tid=30377]]], state=connected] 2026-02-22T20:24:07.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:24:07.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2026-02-22T20:24:07.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:07.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:07.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:07.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2026-02-22T20:24:07.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:07.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:07.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:07.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:07.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_46.status 2026-02-22T20:24:07.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_46.status 2026-02-22T20:24:07.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:07.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2026-02-22T20:24:07.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:24:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:24:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=28, tag_tid=1, entry_tid=30380], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378], [object_number=29, tag_tid=1, entry_tid=30377]]], state=connected] 2026-02-22T20:24:07.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:07.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:07.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:07.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=28, tag_tid=1, entry_tid=30380' '!=' 'object_number=28, tag_tid=1, entry_tid=30380' 2026-02-22T20:24:07.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:07.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_46 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_46 ']' 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_46' 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:08.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:08.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:08.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:08.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:08.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:08.665 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:08.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:08.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:08.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:08.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:08.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:08.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:08.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:08.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-22T20:24:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:08.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:08.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:08.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:08.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:08.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:08.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:08.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2026-02-22T20:24:08.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_46 2026-02-22T20:24:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_46 '' 2026-02-22T20:24:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:08.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_46.status 2026-02-22T20:24:08.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_46.status 2026-02-22T20:24:08.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:08.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:08.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2026-02-22T20:24:08.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:08.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:24:08.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:24:08.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:08.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:08.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:08.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:08.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:24:08.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2026-02-22T20:24:08.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:08.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:08.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:08.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2026-02-22T20:24:08.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:08.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:08.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:24:08.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:08.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_46.status 2026-02-22T20:24:08.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_46.status 2026-02-22T20:24:08.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:08.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:08.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2026-02-22T20:24:08.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 7 2026-02-22T20:24:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 7 2026-02-22T20:24:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=1], [object_number=28, tag_tid=2, entry_tid=0], [object_number=31, tag_tid=1, entry_tid=30379], [object_number=30, tag_tid=1, entry_tid=30378]]], state=connected] 2026-02-22T20:24:08.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.status 2026-02-22T20:24:08.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:24:08.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=29, tag_tid=2, entry_tid=1' -a 'object_number=29, tag_tid=2, entry_tid=1' = 'object_number=29, tag_tid=2, entry_tid=1' 2026-02-22T20:24:08.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:08.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2026-02-22T20:24:08.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:08.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:08.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:08.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:08.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:08.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:24:08.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:09.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2026-02-22T20:24:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_46': 2026-02-22T20:24:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:24:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:24:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:24:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 1127d6f3d2e5 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.1127d6f3d2e5 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:13:46 2026 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:13:46 2026 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:14:47 2026 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 1127d6f3d2e5 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:24:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:24:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: c68fe382-0016-41fb-ae56-94791a9f4771 2026-02-22T20:24:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:24:09.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:09.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_46 snap 2026-02-22T20:24:09.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:09.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:24:09.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:09.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:24:09.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.export 2026-02-22T20:24:09.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_46.export 2026-02-22T20:24:09.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_46.export 2026-02-22T20:24:09.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.export 2026-02-22T20:24:10.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:10.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_46.export 2026-02-22T20:24:10.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:10.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_46.export 2026-02-22T20:24:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_46.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_46.export 2026-02-22T20:24:10.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:10.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:10.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_47 2026-02-22T20:24:10.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_47 snap 2026-02-22T20:24:10.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:10.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:24:11.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2026-02-22T20:24:11.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:11.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:11.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:11.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2026-02-22T20:24:11.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:11.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:11.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:11.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:24:11.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:11.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:11.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:12.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2026-02-22T20:24:12.883 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:12.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:12.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:12.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:24:12.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:24:12.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:24:12.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:24:12.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2026-02-22T20:24:12.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:24:12.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:12.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:12.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:24:12.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:24:12.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:12.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:24:12.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:12.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:24:12.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:12.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2026-02-22T20:24:12.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:24:12.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:12.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:12.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:12.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:12.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:24:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2026-02-22T20:24:12.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:12.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:12.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:12.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:12.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2026-02-22T20:24:12.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:24:12.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:12.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:12.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:12.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:12.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:24:12.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:24:12.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:12.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:12.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:12.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:12.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2026-02-22T20:24:13.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:13.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:13.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:13.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:13.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2026-02-22T20:24:13.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:13.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:24:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:24:13.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:13.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:13.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:13.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:13.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2026-02-22T20:24:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:24:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:24:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:24:13.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:24:13.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_47 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:24:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:24:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:13.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_47 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_47 ']' 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_47' 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:13.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:13.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:13.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:13.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:13.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:13.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:13.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:13.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:13.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2026-02-22T20:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:13.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:13.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:13.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:13.425 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:13.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:13.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:13.476 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:13.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:13.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:13.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:13.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:13.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:13.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_47 2026-02-22T20:24:13.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:13.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:13.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '' 2026-02-22T20:24:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:13.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:13.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:13.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:13.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:13.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=1, entry_tid=53682], [object_number=49, tag_tid=1, entry_tid=53681], [object_number=48, tag_tid=1, entry_tid=53680], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:13.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:13.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=49, tag_tid=2, entry_tid=1' 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:13.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:13.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:13.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:13.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:13.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:13.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:13.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:13.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:13.629 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=1, entry_tid=53682], [object_number=49, tag_tid=1, entry_tid=53681], [object_number=48, tag_tid=1, entry_tid=53680], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:13.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:13.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:13.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=49, tag_tid=2, entry_tid=1' -a 'object_number=49, tag_tid=2, entry_tid=1' = 'object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:13.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=1, entry_tid=53682' '!=' '' 2026-02-22T20:24:13.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:24:13.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=1, entry_tid=53682' = '' 2026-02-22T20:24:13.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:13.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=1' 2026-02-22T20:24:13.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:24:13.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:13.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:24:13.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:13.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:24:13.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:13.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:24:13.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=49, tag_tid=2, entry_tid=1' 2026-02-22T20:24:13.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:24:13.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:13.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:24:13.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:13.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:24:13.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:13.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=53682 2026-02-22T20:24:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 53682 2026-02-22T20:24:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:24:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_47 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_47 ']' 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_47' 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:13.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:13.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:13.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:13.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:13.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:13.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:13.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:13.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:13.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2026-02-22T20:24:13.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:13.915 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:13.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:13.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:13.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:13.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:13.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:13.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:13.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:13.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:13.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:14.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_47 2026-02-22T20:24:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '' 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:14.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:14.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:14.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:14.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=1, entry_tid=53682], [object_number=49, tag_tid=1, entry_tid=53681], [object_number=48, tag_tid=1, entry_tid=53680], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=49, tag_tid=2, entry_tid=1' 2026-02-22T20:24:14.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2026-02-22T20:24:14.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2026-02-22T20:24:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:14.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:14.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:14.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:14.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:14.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=1, entry_tid=53682], [object_number=49, tag_tid=1, entry_tid=53681], [object_number=48, tag_tid=1, entry_tid=53680], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:14.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:14.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=49, tag_tid=2, entry_tid=1' -a 'object_number=49, tag_tid=2, entry_tid=1' = 'object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:14.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=1, entry_tid=53682' '!=' 'object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:14.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:14.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:24:14.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_47 2026-02-22T20:24:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_47 ']' 2026-02-22T20:24:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_47' 2026-02-22T20:24:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2026-02-22T20:24:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:14.555 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:14.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:14.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:14.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:14.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:14.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:14.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:14.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:14.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:14.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:14.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:14.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:14.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:14.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:14.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:14.698 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:14.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:14.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_47 2026-02-22T20:24:14.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:14.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:14.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:14.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '' 2026-02-22T20:24:14.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:14.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:14.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:14.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:14.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:14.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:14.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:14.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:14.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:14.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:14.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:14.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:14.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=1, entry_tid=53682], [object_number=49, tag_tid=1, entry_tid=53681], [object_number=48, tag_tid=1, entry_tid=53680], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:14.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.792 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=49, tag_tid=2, entry_tid=1' 2026-02-22T20:24:14.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2026-02-22T20:24:14.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:14.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:14.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:14.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2026-02-22T20:24:14.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:14.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:14.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:14.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:14.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:14.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:14.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:14.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:14.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:14.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:14.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:14.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=1, entry_tid=53682], [object_number=49, tag_tid=1, entry_tid=53681], [object_number=48, tag_tid=1, entry_tid=53680], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:14.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=49, tag_tid=2, entry_tid=1' -a 'object_number=49, tag_tid=2, entry_tid=1' = 'object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=1, entry_tid=53682' '!=' 'object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_47 2026-02-22T20:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_47 ']' 2026-02-22T20:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_47' 2026-02-22T20:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2026-02-22T20:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2026-02-22T20:24:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:15.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:15.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:15.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:15.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:15.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:15.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:15.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:15.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:15.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2026-02-22T20:24:15.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:15.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:15.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:15.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:15.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:15.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:15.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:15.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:15.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:15.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:15.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:15.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:15.780 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:15.793 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:15.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_47 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '' 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:15.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:15.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:15.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:15.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:15.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:15.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:15.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:15.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:15.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:15.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:15.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=1, entry_tid=53682], [object_number=49, tag_tid=1, entry_tid=53681], [object_number=48, tag_tid=1, entry_tid=53680], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:15.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:15.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=49, tag_tid=2, entry_tid=1' 2026-02-22T20:24:15.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2026-02-22T20:24:15.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:15.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:15.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2026-02-22T20:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:15.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:15.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:15.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:15.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:15.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=50, tag_tid=1, entry_tid=53682], [object_number=49, tag_tid=1, entry_tid=53681], [object_number=48, tag_tid=1, entry_tid=53680], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:15.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:15.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:15.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=49, tag_tid=2, entry_tid=1' -a 'object_number=49, tag_tid=2, entry_tid=1' = 'object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:15.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=50, tag_tid=1, entry_tid=53682' '!=' 'object_number=50, tag_tid=1, entry_tid=53682' 2026-02-22T20:24:15.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:15.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_47 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_47 ']' 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_47' 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:17.552 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:17.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:17.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:17.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:17.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:17.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:17.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:17.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:17.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:17.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:17.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:17.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:17.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:17.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:17.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-22T20:24:17.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:17.702 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:17.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2026-02-22T20:24:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_47 2026-02-22T20:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '' 2026-02-22T20:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:17.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:17.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:17.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:17.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:17.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:17.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:17.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:17.811 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:17.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:17.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=49, tag_tid=2, entry_tid=1' 2026-02-22T20:24:17.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_47.status 2026-02-22T20:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_47.status 2026-02-22T20:24:17.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:17.818 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2026-02-22T20:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 12 2026-02-22T20:24:18.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 12 2026-02-22T20:24:18.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:18.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:18.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=49, tag_tid=2, entry_tid=1], [object_number=48, tag_tid=2, entry_tid=0], [object_number=50, tag_tid=1, entry_tid=53682], [object_number=51, tag_tid=1, entry_tid=53679]]], state=connected] 2026-02-22T20:24:18.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.status 2026-02-22T20:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=49, tag_tid=2, entry_tid=1' 2026-02-22T20:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=49, tag_tid=2, entry_tid=1' -a 'object_number=49, tag_tid=2, entry_tid=1' = 'object_number=49, tag_tid=2, entry_tid=1' 2026-02-22T20:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2026-02-22T20:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:18.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:18.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:24:18.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2026-02-22T20:24:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_47': 2026-02-22T20:24:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:24:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:24:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:24:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 11e394442ef0 2026-02-22T20:24:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.11e394442ef0 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:13:55 2026 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:13:55 2026 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:15:57 2026 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 11e394442ef0 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: fc051830-956f-4240-859a-17e55f5caa96 2026-02-22T20:24:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:24:19.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:19.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_47 snap 2026-02-22T20:24:19.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:19.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:24:19.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:19.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:24:19.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.export 2026-02-22T20:24:19.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_47.export 2026-02-22T20:24:19.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_47.export 2026-02-22T20:24:19.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.export 2026-02-22T20:24:19.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:19.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_47.export 2026-02-22T20:24:19.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:19.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_47.export 2026-02-22T20:24:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_47.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_47.export 2026-02-22T20:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_48 2026-02-22T20:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_48 snap 2026-02-22T20:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:24:22.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2026-02-22T20:24:22.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:22.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:22.901 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:22.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2026-02-22T20:24:22.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:22.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:22.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:22.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:24:22.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:22.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:22.902 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:23.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2026-02-22T20:24:23.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:23.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:24:23.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:24:23.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:23.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:24:23.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:23.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:24:23.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:23.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2026-02-22T20:24:23.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:24:23.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:23.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:23.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:23.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:24:23.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:24:23.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:23.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:23.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:23.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:23.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2026-02-22T20:24:23.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:23.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:23.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:24:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:24:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:23.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2026-02-22T20:24:24.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:24.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:24.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:24.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:24.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:24.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:24.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2026-02-22T20:24:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_48 2026-02-22T20:24:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:24:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:24.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:24.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:24.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:24.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_48 2026-02-22T20:24:24.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:24:24.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:24:24.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:24:24.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:24:24.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:24:24.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:24:24.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:24:24.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2026-02-22T20:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_48 2026-02-22T20:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:24:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:24:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:24:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_48 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_48 ']' 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_48' 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:24.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2026-02-22T20:24:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:24.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:24.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:24.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:24.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:24.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:24.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:24.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:24.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:24.408 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:24.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:24.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:24.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:24.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:24.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:24.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:24.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:24.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2026-02-22T20:24:24.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:24.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:24.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:24.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:24.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:24.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:24.475 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:24.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:24.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:24.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:24.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:24.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:24.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:24.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:24.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:24.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:24.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:24.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:24.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:24.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:24.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:24.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_48 2026-02-22T20:24:24.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:24:24.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:24.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:24:24.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:24.621 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:24.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:24.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:24.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:24.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:24.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:24.631 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:24.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:24.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_48 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '' 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:24.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:24.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:24.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:24.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:24.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:24.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:24.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:24.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:24.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:24.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:24.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:24.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:24.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:24.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:24.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:24.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=26570' '!=' '' 2026-02-22T20:24:24.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:24:24.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=26570' = '' 2026-02-22T20:24:24.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:24.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:24.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:24:24.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:24.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:24:24.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:24.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:24:24.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:24:24.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:24.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:24:24.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:24.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:24:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:24:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=26570 2026-02-22T20:24:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 26570 2026-02-22T20:24:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:24:24.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:24.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:24:24.979 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_48 2026-02-22T20:24:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_48 ']' 2026-02-22T20:24:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_48' 2026-02-22T20:24:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2026-02-22T20:24:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:24.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:24.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:24.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:24.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:24.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:24.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:24.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:24.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:24.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:24.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:24.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:24.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:25.059 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:25.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:25.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2026-02-22T20:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:25.072 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:25.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:25.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:25.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:25.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:25.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:25.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:25.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:25.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2026-02-22T20:24:25.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:25.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:25.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:25.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:25.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:25.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:25.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:25.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:25.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:25.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:25.203 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:25.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:25.204 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:25.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:25.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_48 2026-02-22T20:24:25.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:24:25.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:25.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:25.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:25.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:25.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:25.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:25.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:25.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_48 2026-02-22T20:24:25.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:25.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '' 2026-02-22T20:24:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:25.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:25.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:25.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:25.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:25.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:25.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2026-02-22T20:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2026-02-22T20:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:25.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:25.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:25.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:25.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:25.329 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:25.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:25.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:25.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:25.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:25.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:25.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:25.366 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:25.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:25.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:25.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:25.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=26570' '!=' 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:25.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:25.371 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_48 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_48 ']' 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_48' 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:25.772 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:25.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:25.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:25.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:25.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:25.773 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:25.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:25.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:25.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:25.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:25.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:25.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:25.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:25.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:25.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:25.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:25.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:25.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:25.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:25.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:25.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:25.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_48 2026-02-22T20:24:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:24:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:24:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:25.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:26.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:26.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:26.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:26.001 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_48 2026-02-22T20:24:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '' 2026-02-22T20:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:26.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:26.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:26.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:26.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:26.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:26.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:26.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2026-02-22T20:24:26.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:26.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:26.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:26.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2026-02-22T20:24:26.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:26.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:26.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:26.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:26.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:26.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:26.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:26.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:26.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:26.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:26.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:26.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:26.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:26.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:26.130 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:26.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:26.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:26.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=26570' '!=' 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:26.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:26.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:24:26.838 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6269624469931011) > weight (0.25) 2026-02-22T20:24:26.838 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.33771651477151987) > weight (0.25) 2026-02-22T20:24:26.838 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2530360382756487) > weight (0.25) 2026-02-22T20:24:26.839 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4785613148882496) > weight (0.25) 2026-02-22T20:24:26.839 INFO:tasks.rbd_mirror_thrash:waiting for 91.05601299919796 secs before thrashing 2026-02-22T20:24:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_48 2026-02-22T20:24:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_48 ']' 2026-02-22T20:24:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_48' 2026-02-22T20:24:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2026-02-22T20:24:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:26.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:26.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:26.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:26.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:26.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:26.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:26.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:26.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:26.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:26.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2026-02-22T20:24:26.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:26.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:26.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:26.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:26.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:26.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:26.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:26.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:26.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:27.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:27.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:27.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:27.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:27.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2026-02-22T20:24:27.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:27.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:27.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:27.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:27.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:27.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:27.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:27.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:27.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:27.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:27.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:27.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:27.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:27.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:27.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:27.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_48 2026-02-22T20:24:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:24:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:24:27.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:27.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:27.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:27.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:27.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:27.176 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_48 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '' 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:27.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:27.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:27.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:27.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:27.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:27.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:27.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:27.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:27.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:27.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:27.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:27.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:27.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:27.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:27.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:27.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:27.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:27.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=26570' '!=' 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:27.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:27.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:24:28.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_48 2026-02-22T20:24:28.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:28.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_48 ']' 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_48' 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:28.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:28.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:28.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:28.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:28.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:28.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:28.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:28.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:28.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:28.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:28.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:28.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:28.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:28.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:28.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2026-02-22T20:24:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:29.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:29.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:29.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:29.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:29.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:29.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:29.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_48 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:29.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:29.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:29.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:29.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:29.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_48 2026-02-22T20:24:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '' 2026-02-22T20:24:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:29.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:29.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:29.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:29.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:29.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:29.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:29.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:29.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:29.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:29.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:29.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:29.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:29.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:29.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:29.265 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:29.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:29.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:29.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=26570' '!=' 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:29.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:29.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:24:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_48 2026-02-22T20:24:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_48 ']' 2026-02-22T20:24:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_48' 2026-02-22T20:24:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2026-02-22T20:24:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2026-02-22T20:24:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:31.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:31.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:31.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:31.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:31.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:31.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:31.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:31.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:31.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:31.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:31.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:31.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:31.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:31.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:31.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:31.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:31.286 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:31.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:31.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:31.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2026-02-22T20:24:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:31.353 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:31.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:31.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:31.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:31.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:31.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2026-02-22T20:24:31.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:31.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:31.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:31.428 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:31.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:31.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:31.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:31.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:31.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:31.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:31.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:31.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:31.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:31.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:31.497 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:31.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_48 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:31.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:31.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:31.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:31.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:31.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:31.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_48 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '' 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:31.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:31.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:31.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:31.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:31.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:31.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:31.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:31.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2026-02-22T20:24:31.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:31.604 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2026-02-22T20:24:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:31.605 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:31.606 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:31.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:31.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:31.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:31.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=26, tag_tid=1, entry_tid=26570], [object_number=25, tag_tid=1, entry_tid=26569], [object_number=24, tag_tid=1, entry_tid=26568], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:31.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=26, tag_tid=1, entry_tid=26570' '!=' 'object_number=26, tag_tid=1, entry_tid=26570' 2026-02-22T20:24:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:31.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:24:33.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_48 2026-02-22T20:24:33.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:33.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:33.645 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:33.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:33.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_48 ']' 2026-02-22T20:24:33.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_48' 2026-02-22T20:24:33.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2026-02-22T20:24:33.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:33.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:33.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:33.646 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:33.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:33.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:33.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:33.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:33.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:33.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:33.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2026-02-22T20:24:33.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:33.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:33.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:33.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:33.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:33.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:33.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:33.649 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:33.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:33.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:33.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:33.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:33.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:33.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:33.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:33.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:33.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:33.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:33.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:33.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2026-02-22T20:24:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:33.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:33.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:33.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:33.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:33.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:33.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:33.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:33.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:33.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:33.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:33.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-22T20:24:33.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:33.847 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_48 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:33.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:33.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:33.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:33.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_48 2026-02-22T20:24:33.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_48 2026-02-22T20:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '' 2026-02-22T20:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:33.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:33.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:33.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:33.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:33.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:33.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:33.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:33.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:33.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:33.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:33.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:33.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2026-02-22T20:24:33.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_48.status 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_48.status 2026-02-22T20:24:33.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:33.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:33.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2026-02-22T20:24:33.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 6 2026-02-22T20:24:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 6 2026-02-22T20:24:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=1], [object_number=24, tag_tid=2, entry_tid=0], [object_number=26, tag_tid=1, entry_tid=26570], [object_number=27, tag_tid=1, entry_tid=26567]]], state=connected] 2026-02-22T20:24:34.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.status 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=25, tag_tid=2, entry_tid=1' -a 'object_number=25, tag_tid=2, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=1' 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:24:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:35.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_48': 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 120817503de2 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.120817503de2 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:14:09 2026 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:14:09 2026 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:15:10 2026 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 120817503de2 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: fa8d2945-f6ff-44ff-8d16-6a98b0fcc738 2026-02-22T20:24:35.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_48 snap 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.export 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_48.export 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_48.export 2026-02-22T20:24:35.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.export 2026-02-22T20:24:35.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:35.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_48.export 2026-02-22T20:24:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_48.export 2026-02-22T20:24:35.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_48.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_48.export 2026-02-22T20:24:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_49 2026-02-22T20:24:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_49 snap 2026-02-22T20:24:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:24:38.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2026-02-22T20:24:38.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:38.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:38.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:38.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2026-02-22T20:24:38.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:38.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:38.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:38.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:24:38.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:38.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:38.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:39.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:39.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:39.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:39.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2026-02-22T20:24:39.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2026-02-22T20:24:39.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:39.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:24:39.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:40.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2026-02-22T20:24:40.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:24:40.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:24:40.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:24:40.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:24:40.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:24:40.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:24:40.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:24:40.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:40.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2026-02-22T20:24:40.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:40.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:40.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:40.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:40.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_49 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:40.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:24:40.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_49 2026-02-22T20:24:40.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:40.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:40.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:40.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_49 ']' 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_49' 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:40.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:40.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:40.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:40.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2026-02-22T20:24:40.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:40.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:40.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:40.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:40.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:40.273 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:40.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:40.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:40.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:40.276 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:40.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:40.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:40.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:40.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2026-02-22T20:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:40.335 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:40.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:40.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:40.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:40.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:40.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_49 2026-02-22T20:24:40.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:40.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '' 2026-02-22T20:24:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:40.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:40.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:40.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:40.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2026-02-22T20:24:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2026-02-22T20:24:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:40.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:40.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:40.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:40.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:40.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:40.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:40.503 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41158' '!=' '' 2026-02-22T20:24:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ break 2026-02-22T20:24:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41158' = '' 2026-02-22T20:24:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ last_mirror_pos='object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:40.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:24:40.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:40.506 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_tag=2 2026-02-22T20:24:40.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:40.507 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-22T20:24:40.508 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:40.509 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_tag=1 2026-02-22T20:24:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:24:40.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:40.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_entry=1 2026-02-22T20:24:40.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:40.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-22T20:24:40.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut -d= -f 2 2026-02-22T20:24:40.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_entry=41158 2026-02-22T20:24:40.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 2 = 1 -a 1 -le 41158 2026-02-22T20:24:40.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:24:40.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:40.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:24:40.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_49 2026-02-22T20:24:40.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:40.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:40.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:40.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:40.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_49 ']' 2026-02-22T20:24:40.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_49' 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:40.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:40.729 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:40.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2026-02-22T20:24:40.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:40.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:40.784 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:40.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:40.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:40.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:40.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:40.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:40.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:40.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:40.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_49 2026-02-22T20:24:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '' 2026-02-22T20:24:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:40.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:40.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:40.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:40.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:40.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:40.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:40.910 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:40.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:40.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:40.913 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:40.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:40.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:40.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:40.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:40.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:40.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:40.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:40.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41158' '!=' 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:40.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:40.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.4 2026-02-22T20:24:41.354 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_49 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_49 ']' 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_49' 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:41.355 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:41.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:41.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:41.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:41.356 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2026-02-22T20:24:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:41.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:41.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:41.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:41.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:41.370 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2026-02-22T20:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:41.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:41.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:41.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:41.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:41.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:41.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:41.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_49 2026-02-22T20:24:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '' 2026-02-22T20:24:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:41.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:41.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:41.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:41.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:41.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2026-02-22T20:24:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:41.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2026-02-22T20:24:41.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:41.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:41.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:41.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:41.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:41.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:41.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:41.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:41.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:41.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:41.582 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:41.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:41.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:41.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:41.583 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41158' '!=' 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.8 2026-02-22T20:24:42.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_49 2026-02-22T20:24:42.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:42.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_49 ']' 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_49' 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:42.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:42.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2026-02-22T20:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:42.405 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:42.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:42.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:42.406 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:42.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2026-02-22T20:24:42.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:42.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2026-02-22T20:24:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:42.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:42.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_49 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '' 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:42.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:42.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:42.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:42.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:42.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:42.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:42.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:42.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:42.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2026-02-22T20:24:42.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:42.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:42.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:42.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2026-02-22T20:24:42.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:42.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:42.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:42.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:42.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:42.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:42.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:42.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:42.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:42.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41158' '!=' 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1.6 2026-02-22T20:24:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_49 2026-02-22T20:24:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_49 ']' 2026-02-22T20:24:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_49' 2026-02-22T20:24:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2026-02-22T20:24:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2026-02-22T20:24:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:44.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:44.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:44.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:44.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:44.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:44.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:44.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2026-02-22T20:24:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:44.375 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_49 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '' 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:44.376 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:44.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:44.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:44.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:44.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:44.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:44.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:44.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:44.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2026-02-22T20:24:44.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:44.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:44.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:44.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2026-02-22T20:24:44.422 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:44.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:44.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:44.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:44.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:44.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:44.423 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:44.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:44.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:44.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:44.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:44.458 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:44.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:44.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41158' '!=' 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:24:46.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_49 2026-02-22T20:24:46.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:46.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:46.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:46.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:46.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_49 ']' 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_49' 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:46.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2026-02-22T20:24:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:46.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:46.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:46.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:46.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:46.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:46.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2026-02-22T20:24:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:46.542 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:46.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:46.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:46.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:46.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:46.556 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_49 2026-02-22T20:24:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '' 2026-02-22T20:24:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:46.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:46.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:46.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:46.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:46.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:46.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:46.650 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:46.651 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2026-02-22T20:24:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2026-02-22T20:24:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:46.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:46.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:46.682 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:46.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:46.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=38, tag_tid=1, entry_tid=41158], [object_number=37, tag_tid=1, entry_tid=41157], [object_number=36, tag_tid=1, entry_tid=41156], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:46.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:46.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:46.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:46.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 'object_number=38, tag_tid=1, entry_tid=41158' '!=' 'object_number=38, tag_tid=1, entry_tid=41158' 2026-02-22T20:24:46.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:46.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:24:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_49 2026-02-22T20:24:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_49 ']' 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_49' 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:48.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2026-02-22T20:24:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:48.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:48.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2026-02-22T20:24:48.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:48.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2026-02-22T20:24:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:48.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:48.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_49 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '' 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:48.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:48.850 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:48.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2026-02-22T20:24:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2026-02-22T20:24:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:48.884 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:48.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:24:48.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:48.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_49.status 2026-02-22T20:24:48.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_49.status 2026-02-22T20:24:48.885 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:48.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:48.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2026-02-22T20:24:48.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:48.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 9 2026-02-22T20:24:48.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 9 2026-02-22T20:24:48.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:48.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:48.914 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=37, tag_tid=2, entry_tid=1], [object_number=36, tag_tid=2, entry_tid=0], [object_number=38, tag_tid=1, entry_tid=41158], [object_number=39, tag_tid=1, entry_tid=41155]]], state=connected] 2026-02-22T20:24:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.status 2026-02-22T20:24:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=37, tag_tid=2, entry_tid=1' -a 'object_number=37, tag_tid=2, entry_tid=1' = 'object_number=37, tag_tid=2, entry_tid=1' 2026-02-22T20:24:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2026-02-22T20:24:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:48.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:48.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:48.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:48.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:24:48.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2026-02-22T20:24:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_49': 2026-02-22T20:24:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:24:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:24:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:24:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 10aaf1ee4cac 2026-02-22T20:24:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.10aaf1ee4cac 2026-02-22T20:24:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:14:18 2026 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:14:18 2026 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:15:19 2026 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 10aaf1ee4cac 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: bde201af-aae0-4711-81e1-e7956e7d56ac 2026-02-22T20:24:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:24:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_49 snap 2026-02-22T20:24:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:49.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:24:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:24:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.export 2026-02-22T20:24:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_49.export 2026-02-22T20:24:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_49.export 2026-02-22T20:24:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.export 2026-02-22T20:24:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_49.export 2026-02-22T20:24:50.279 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_49.export 2026-02-22T20:24:50.338 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_49.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_49.export 2026-02-22T20:24:50.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:50.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:50.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_50 2026-02-22T20:24:50.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_snap cluster2 mirror image_50 snap 2026-02-22T20:24:50.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:50.384 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:50.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:24:50.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:50.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2026-02-22T20:24:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2026-02-22T20:24:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:24:51.931 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2026-02-22T20:24:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:24:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=started 2026-02-22T20:24:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:51.932 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:52.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2026-02-22T20:24:52.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:52.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:52.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:24:52.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=started 2026-02-22T20:24:52.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local status_result 2026-02-22T20:24:52.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=stopped 2026-02-22T20:24:52.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2026-02-22T20:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -i state 2026-02-22T20:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_instance=cluster1 2026-02-22T20:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance=cluster1 2026-02-22T20:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local loop_instance 2026-02-22T20:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sleep 0 2026-02-22T20:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ seq 0 3 2026-02-22T20:24:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:0 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:24:52.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:0 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=0 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=0 2026-02-22T20:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:52.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2026-02-22T20:24:53.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2026-02-22T20:24:53.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:53.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:53.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2026-02-22T20:24:53.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:1 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:1 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:53.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:53.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=1 2026-02-22T20:24:53.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=1 2026-02-22T20:24:53.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2026-02-22T20:24:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2026-02-22T20:24:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:2 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:2 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=2 2026-02-22T20:24:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=2 2026-02-22T20:24:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:53.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:53.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2026-02-22T20:24:53.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2026-02-22T20:24:53.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:24:53.154 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1:3 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ shift 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=cluster 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval cluster=cluster1 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster=cluster1 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local val=cluster1:3 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local instance_var_name=instance 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ cluster=cluster1 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ local_cluster=cluster1 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ eval instance=3 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ instance=3 2026-02-22T20:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2026-02-22T20:24:53.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ return 0 2026-02-22T20:24:53.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:24:53.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:24:53.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ grep -i Replaying 2026-02-22T20:24:53.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: "state": "Replaying" 2026-02-22T20:24:53.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=started 2026-02-22T20:24:53.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test started = started 2026-02-22T20:24:53.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' journal = journal ']' 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_50 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local local_cluster=cluster1 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:24:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-22T20:24:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-22T20:24:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:24:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:24:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.2 2026-02-22T20:24:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ flush cluster1 mirror image_50 2026-02-22T20:24:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:24:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T20:24:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n image_50 ']' 2026-02-22T20:24:53.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmd='rbd mirror flush mirror/image_50' 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_instance=cluster1 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance=cluster1 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loop_instance 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:24:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:24:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2026-02-22T20:24:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:24:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:24:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:0 2026-02-22T20:24:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:24:53.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:0 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=0 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=0 2026-02-22T20:24:53.444 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:24:53.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:53.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:53.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:24:53.456 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2026-02-22T20:24:53.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2026-02-22T20:24:53.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:53.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:53.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:1 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=1 2026-02-22T20:24:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=1 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:1 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:53.521 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=1 2026-02-22T20:24:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:24:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=1 2026-02-22T20:24:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=1 2026-02-22T20:24:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:24:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:24:53.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2026-02-22T20:24:53.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2026-02-22T20:24:53.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:53.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:53.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:53.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_50 2026-02-22T20:24:53.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:24:53.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:53.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:53.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:2 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=2 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=2 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:2 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=2 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=2 2026-02-22T20:24:53.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=2 2026-02-22T20:24:53.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:24:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:53.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:24:53.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_50 2026-02-22T20:24:53.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2026-02-22T20:24:53.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd mirror flush : flush rbd mirror 2026-02-22T20:24:53.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:admin_socket: invalid command 2026-02-22T20:24:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:24:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_50 2026-02-22T20:24:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1:3 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=3 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=3 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local val=cluster1:3 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local instance_var_name=instance 2026-02-22T20:24:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:24:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=3 2026-02-22T20:24:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:24:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval local_cluster=cluster1 2026-02-22T20:24:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ local_cluster=cluster1 2026-02-22T20:24:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ eval instance=3 2026-02-22T20:24:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ instance=3 2026-02-22T20:24:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:24:53.680 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:53.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:53.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:24:53.681 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_50 2026-02-22T20:24:53.751 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_master_journal_position cluster2 mirror image_50 2026-02-22T20:24:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_50 2026-02-22T20:24:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_50 '' 2026-02-22T20:24:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_50 2026-02-22T20:24:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local id_regexp= 2026-02-22T20:24:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_50.status 2026-02-22T20:24:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_50.status 2026-02-22T20:24:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.status 2026-02-22T20:24:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2026-02-22T20:24:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.status 2026-02-22T20:24:53.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:24:53.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:24:53.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:53.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=34819], [object_number=34, tag_tid=1, entry_tid=34818]]], state=connected] 2026-02-22T20:24:53.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=34819], [object_number=34, tag_tid=1, entry_tid=34818]]], state=connected] 2026-02-22T20:24:53.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.status 2026-02-22T20:24:53.795 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ master_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_50 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_50 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local 'id_regexp=..*' 2026-02-22T20:24:53.796 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ mkfname cluster2-mirror-image_50.status 2026-02-22T20:24:53.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ echo cluster2-mirror-image_50.status 2026-02-22T20:24:53.797 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-22T20:24:53.798 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local status_log=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.status 2026-02-22T20:24:53.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2026-02-22T20:24:53.799 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ tee /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.status 2026-02-22T20:24:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:minimum_set: 8 2026-02-22T20:24:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:active_set: 8 2026-02-22T20:24:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:registered clients: 2026-02-22T20:24:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=34819], [object_number=34, tag_tid=1, entry_tid=34818]]], state=connected] 2026-02-22T20:24:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: [id=31e413e5-c6f1-4547-9d42-a0d4670a12a2, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=1], [object_number=32, tag_tid=2, entry_tid=0], [object_number=35, tag_tid=1, entry_tid=34819], [object_number=34, tag_tid=1, entry_tid=34818]]], state=connected] 2026-02-22T20:24:53.829 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.status 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 'object_number=33, tag_tid=2, entry_tid=1' -a 'object_number=33, tag_tid=2, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=1' 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:24:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:24:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2026-02-22T20:24:55.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_50': 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 128 MiB in 32 objects 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 1 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 112787d5fce4 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.112787d5fce4 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:24:55.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:14:31 2026 2026-02-22T20:24:55.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:14:31 2026 2026-02-22T20:24:55.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:15:32 2026 2026-02-22T20:24:55.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: protected: False 2026-02-22T20:24:55.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 112787d5fce4 2026-02-22T20:24:55.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:24:55.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:24:55.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: f5d1390e-2f7b-49b3-bf9c-122d3f3825d7 2026-02-22T20:24:55.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: false 2026-02-22T20:24:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ compare_image_snaps mirror image_50 snap 2026-02-22T20:24:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:24:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap_name=snap 2026-02-22T20:24:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local ret=0 2026-02-22T20:24:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local rmt_export=/tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.export 2026-02-22T20:24:55.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local loc_export=/tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_50.export 2026-02-22T20:24:55.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_50.export 2026-02-22T20:24:55.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.export 2026-02-22T20:24:55.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:55.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_50.export 2026-02-22T20:24:55.430 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:24:55.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cmp /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_50.export 2026-02-22T20:24:55.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -f /tmp/tmp.Xb7Kt1LMAh/cluster2-mirror-image_50.export /tmp/tmp.Xb7Kt1LMAh/cluster1-mirror-image_50.export 2026-02-22T20:24:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ testlog 'TEST: delete many images' 2026-02-22T20:24:55.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ tee -a /tmp/tmp.Xb7Kt1LMAh/rbd-mirror.test.log 2026-02-22T20:24:55.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ date '+%F %T' 2026-02-22T20:24:55.529 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo 2026-02-22 20:24:55 TEST: delete many images 2026-02-22T20:24:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:2026-02-22 20:24:55 TEST: delete many images 2026-02-22T20:24:55.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 50 2026-02-22T20:24:55.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:55.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_1 2026-02-22T20:24:55.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2026-02-22T20:24:55.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:55.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:55.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:24:55.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:24:55.532 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2026-02-22T20:24:55.923 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_1 2026-02-22T20:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:55.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:24:55.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:24:55.938 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:55.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_1 2026-02-22T20:24:55.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:55.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:55.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:24:55.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2026-02-22T20:24:55.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2026-02-22T20:24:56.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:24:56.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:56.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:56.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_2 2026-02-22T20:24:56.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2026-02-22T20:24:56.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:56.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:56.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:24:56.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:24:56.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2026-02-22T20:24:56.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:24:56.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_2 2026-02-22T20:24:56.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:56.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:56.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:24:56.942 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:24:56.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:56.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_2 2026-02-22T20:24:56.943 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:56.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:56.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:24:56.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2026-02-22T20:24:56.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2026-02-22T20:24:57.118 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:24:57.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:57.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:57.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_3 2026-02-22T20:24:57.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2026-02-22T20:24:57.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:57.124 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:57.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:24:57.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:24:57.125 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2026-02-22T20:24:57.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:24:57.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_3 2026-02-22T20:24:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:24:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:24:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_3 2026-02-22T20:24:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:57.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:24:57.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2026-02-22T20:24:57.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2026-02-22T20:24:58.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:24:58.119 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:58.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:58.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_4 2026-02-22T20:24:58.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2026-02-22T20:24:58.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:58.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:58.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:24:58.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:24:58.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2026-02-22T20:24:58.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:24:58.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_4 2026-02-22T20:24:58.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:58.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:58.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:24:58.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:24:58.951 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:58.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_4 2026-02-22T20:24:58.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:58.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:58.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:24:58.952 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2026-02-22T20:24:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2026-02-22T20:24:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:24:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:24:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:24:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_5 2026-02-22T20:24:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2026-02-22T20:24:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:24:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:24:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2026-02-22T20:24:59.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:24:59.956 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_5 2026-02-22T20:24:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:24:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:24:59.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:24:59.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_5 2026-02-22T20:24:59.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:24:59.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:24:59.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:24:59.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2026-02-22T20:24:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2026-02-22T20:25:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:00.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:00.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:00.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_6 2026-02-22T20:25:00.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2026-02-22T20:25:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:25:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2026-02-22T20:25:00.944 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:00.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_6 2026-02-22T20:25:00.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:00.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:00.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:25:00.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:00.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:00.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_6 2026-02-22T20:25:00.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:00.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:00.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:25:00.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2026-02-22T20:25:00.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2026-02-22T20:25:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:01.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:01.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:01.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_7 2026-02-22T20:25:01.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2026-02-22T20:25:01.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:01.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:01.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:25:01.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:01.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2026-02-22T20:25:01.945 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:01.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_7 2026-02-22T20:25:01.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:01.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:01.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:25:01.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:01.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:01.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_7 2026-02-22T20:25:01.962 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:01.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:01.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:25:01.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2026-02-22T20:25:01.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2026-02-22T20:25:02.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_8 2026-02-22T20:25:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2026-02-22T20:25:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:25:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2026-02-22T20:25:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_8 2026-02-22T20:25:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:25:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_8 2026-02-22T20:25:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:25:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2026-02-22T20:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2026-02-22T20:25:03.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:03.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:03.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:03.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_9 2026-02-22T20:25:03.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2026-02-22T20:25:03.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:03.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:03.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:25:03.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:03.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2026-02-22T20:25:03.948 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:03.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_9 2026-02-22T20:25:03.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:03.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:03.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:25:03.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:03.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:03.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_9 2026-02-22T20:25:03.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:03.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:03.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:25:03.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2026-02-22T20:25:03.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2026-02-22T20:25:04.728 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_10 2026-02-22T20:25:04.733 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2026-02-22T20:25:04.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:04.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:04.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:25:04.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:04.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2026-02-22T20:25:05.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:05.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_10 2026-02-22T20:25:05.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:05.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:05.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:25:05.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:05.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:05.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_10 2026-02-22T20:25:05.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:05.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:05.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:25:05.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2026-02-22T20:25:05.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2026-02-22T20:25:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_11 2026-02-22T20:25:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2026-02-22T20:25:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:25:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:06.049 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2026-02-22T20:25:06.950 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:06.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_11 2026-02-22T20:25:06.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:06.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:06.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:25:06.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:06.963 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_11 2026-02-22T20:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2026-02-22T20:25:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2026-02-22T20:25:07.128 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:07.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:07.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:07.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_12 2026-02-22T20:25:07.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2026-02-22T20:25:07.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:07.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:07.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:25:07.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:07.134 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2026-02-22T20:25:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:07.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_12 2026-02-22T20:25:07.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:07.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:07.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:25:07.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:07.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_12 2026-02-22T20:25:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:25:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2026-02-22T20:25:08.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2026-02-22T20:25:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_13 2026-02-22T20:25:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2026-02-22T20:25:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:25:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2026-02-22T20:25:08.959 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:08.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_13 2026-02-22T20:25:08.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:08.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:08.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:25:08.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:08.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:08.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_13 2026-02-22T20:25:08.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:08.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:08.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:25:08.977 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2026-02-22T20:25:09.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2026-02-22T20:25:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:09.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:09.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:09.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_14 2026-02-22T20:25:09.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2026-02-22T20:25:09.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:09.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:09.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:25:09.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:09.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2026-02-22T20:25:09.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7682306784014898) > weight (0.25) 2026-02-22T20:25:09.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8397591508356781) > weight (0.25) 2026-02-22T20:25:09.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.44226906462374616) > weight (0.25) 2026-02-22T20:25:09.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8609843996303069) > weight (0.25) 2026-02-22T20:25:09.594 INFO:tasks.rbd_mirror_thrash:waiting for 99.84170875735221 secs before thrashing 2026-02-22T20:25:09.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:09.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_14 2026-02-22T20:25:09.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:09.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:09.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:25:09.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:09.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:09.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_14 2026-02-22T20:25:09.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:09.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:09.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:25:09.861 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2026-02-22T20:25:09.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2026-02-22T20:25:10.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_15 2026-02-22T20:25:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2026-02-22T20:25:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:25:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2026-02-22T20:25:10.958 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:10.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_15 2026-02-22T20:25:10.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:10.971 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:10.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:25:10.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:10.972 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_15 2026-02-22T20:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2026-02-22T20:25:11.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2026-02-22T20:25:11.133 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_16 2026-02-22T20:25:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2026-02-22T20:25:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:25:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2026-02-22T20:25:11.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_16 2026-02-22T20:25:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:25:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_16 2026-02-22T20:25:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:25:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2026-02-22T20:25:12.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2026-02-22T20:25:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_17 2026-02-22T20:25:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2026-02-22T20:25:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:25:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2026-02-22T20:25:12.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:12.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_17 2026-02-22T20:25:12.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:12.984 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:12.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:25:12.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:12.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_17 2026-02-22T20:25:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:25:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2026-02-22T20:25:13.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2026-02-22T20:25:13.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_18 2026-02-22T20:25:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2026-02-22T20:25:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:25:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:13.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2026-02-22T20:25:13.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_18 2026-02-22T20:25:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:25:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_18 2026-02-22T20:25:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:25:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2026-02-22T20:25:14.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2026-02-22T20:25:14.157 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:14.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:14.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:14.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_19 2026-02-22T20:25:14.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2026-02-22T20:25:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:25:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:14.163 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2026-02-22T20:25:14.851 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:14.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_19 2026-02-22T20:25:14.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:14.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:14.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:25:14.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:14.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:14.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_19 2026-02-22T20:25:14.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:14.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:14.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:25:14.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2026-02-22T20:25:14.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2026-02-22T20:25:15.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_20 2026-02-22T20:25:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2026-02-22T20:25:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:25:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:15.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2026-02-22T20:25:15.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_20 2026-02-22T20:25:15.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:15.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:15.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:25:15.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:15.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:15.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_20 2026-02-22T20:25:15.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:15.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:15.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:25:15.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2026-02-22T20:25:16.025 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2026-02-22T20:25:16.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_21 2026-02-22T20:25:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2026-02-22T20:25:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:25:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2026-02-22T20:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_21 2026-02-22T20:25:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:25:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:16.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_21 2026-02-22T20:25:16.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:16.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:16.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:25:16.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2026-02-22T20:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2026-02-22T20:25:17.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:17.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:17.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:17.164 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_22 2026-02-22T20:25:17.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2026-02-22T20:25:17.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:17.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:17.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:25:17.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:17.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2026-02-22T20:25:17.980 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:17.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_22 2026-02-22T20:25:17.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:17.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:17.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:25:17.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:17.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:17.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_22 2026-02-22T20:25:17.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:17.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:17.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:25:17.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2026-02-22T20:25:18.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2026-02-22T20:25:18.175 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:18.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:18.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:18.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_23 2026-02-22T20:25:18.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2026-02-22T20:25:18.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:18.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:18.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:25:18.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:18.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2026-02-22T20:25:18.982 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:18.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_23 2026-02-22T20:25:18.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:18.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:18.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:25:18.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:18.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_23 2026-02-22T20:25:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:25:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2026-02-22T20:25:19.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2026-02-22T20:25:19.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_24 2026-02-22T20:25:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2026-02-22T20:25:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:25:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2026-02-22T20:25:19.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_24 2026-02-22T20:25:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:25:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:19.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_24 2026-02-22T20:25:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:25:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2026-02-22T20:25:19.900 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2026-02-22T20:25:20.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:20.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:20.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:20.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_25 2026-02-22T20:25:20.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2026-02-22T20:25:20.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:20.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:20.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:25:20.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:20.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2026-02-22T20:25:20.987 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:21.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_25 2026-02-22T20:25:21.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:21.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:21.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:25:21.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:21.004 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:21.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_25 2026-02-22T20:25:21.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:25:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2026-02-22T20:25:21.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2026-02-22T20:25:21.099 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:25:21.092+0000 7f557ffc8640 -1 reset not still connected to 0x5592385d6dd0 2026-02-22T20:25:21.099 INFO:tasks.ceph.cluster2.osd.2.trial066.stderr:2026-02-22T20:25:21.092+0000 7f557ffc8640 -1 reset not still connected to 0x55923d98de10 2026-02-22T20:25:21.178 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:21.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:21.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:21.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_26 2026-02-22T20:25:21.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2026-02-22T20:25:21.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:21.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:21.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:25:21.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:21.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2026-02-22T20:25:21.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:22.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_26 2026-02-22T20:25:22.010 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:25:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:22.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:22.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_26 2026-02-22T20:25:22.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:22.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:22.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:25:22.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2026-02-22T20:25:22.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2026-02-22T20:25:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:22.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:22.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:22.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_27 2026-02-22T20:25:22.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2026-02-22T20:25:22.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:22.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:22.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:25:22.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:22.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2026-02-22T20:25:22.992 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_27 2026-02-22T20:25:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:25:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_27 2026-02-22T20:25:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:25:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2026-02-22T20:25:23.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2026-02-22T20:25:23.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_28 2026-02-22T20:25:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2026-02-22T20:25:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:25:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2026-02-22T20:25:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:24.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_28 2026-02-22T20:25:24.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:24.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:24.018 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:25:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_28 2026-02-22T20:25:24.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:24.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:24.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:25:24.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2026-02-22T20:25:24.048 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2026-02-22T20:25:24.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_29 2026-02-22T20:25:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2026-02-22T20:25:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:24.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:25:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2026-02-22T20:25:24.860 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:24.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_29 2026-02-22T20:25:24.874 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:24.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:24.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:25:24.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:24.875 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:24.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_29 2026-02-22T20:25:24.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:24.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:24.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:25:24.876 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2026-02-22T20:25:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2026-02-22T20:25:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_30 2026-02-22T20:25:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2026-02-22T20:25:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:25:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2026-02-22T20:25:26.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_30 2026-02-22T20:25:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:26.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:26.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:25:26.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:26.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_30 2026-02-22T20:25:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:25:26.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2026-02-22T20:25:26.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2026-02-22T20:25:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:26.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:26.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:26.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_31 2026-02-22T20:25:26.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2026-02-22T20:25:26.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:26.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:26.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:25:26.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:26.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2026-02-22T20:25:27.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:27.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_31 2026-02-22T20:25:27.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:27.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:27.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:25:27.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:27.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:27.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_31 2026-02-22T20:25:27.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:27.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:27.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:25:27.023 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2026-02-22T20:25:27.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2026-02-22T20:25:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_32 2026-02-22T20:25:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2026-02-22T20:25:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:25:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2026-02-22T20:25:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_32 2026-02-22T20:25:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:25:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_32 2026-02-22T20:25:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:25:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2026-02-22T20:25:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2026-02-22T20:25:28.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_33 2026-02-22T20:25:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2026-02-22T20:25:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:25:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2026-02-22T20:25:29.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:29.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_33 2026-02-22T20:25:29.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:29.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:29.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:25:29.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:29.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_33 2026-02-22T20:25:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:25:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2026-02-22T20:25:29.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2026-02-22T20:25:29.188 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:29.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:29.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_34 2026-02-22T20:25:29.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2026-02-22T20:25:29.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:29.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:29.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:25:29.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:29.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2026-02-22T20:25:29.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:29.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_34 2026-02-22T20:25:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:25:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:29.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_34 2026-02-22T20:25:29.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:29.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:29.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:25:29.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2026-02-22T20:25:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2026-02-22T20:25:30.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:30.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:30.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:30.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_35 2026-02-22T20:25:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2026-02-22T20:25:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:25:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:30.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2026-02-22T20:25:31.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_35 2026-02-22T20:25:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:25:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_35 2026-02-22T20:25:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:25:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2026-02-22T20:25:31.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2026-02-22T20:25:31.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_36 2026-02-22T20:25:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2026-02-22T20:25:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:25:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:31.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2026-02-22T20:25:32.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:32.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_36 2026-02-22T20:25:32.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:32.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:32.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:25:32.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:32.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_36 2026-02-22T20:25:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:25:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2026-02-22T20:25:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2026-02-22T20:25:32.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:32.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_37 2026-02-22T20:25:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2026-02-22T20:25:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:25:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:32.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2026-02-22T20:25:33.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_37 2026-02-22T20:25:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:25:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_37 2026-02-22T20:25:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:25:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2026-02-22T20:25:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2026-02-22T20:25:33.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:33.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:33.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:33.206 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_38 2026-02-22T20:25:33.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2026-02-22T20:25:33.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:33.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:33.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:25:33.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:33.207 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2026-02-22T20:25:34.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:34.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_38 2026-02-22T20:25:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:25:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:34.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_38 2026-02-22T20:25:34.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:34.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:34.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:25:34.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2026-02-22T20:25:34.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2026-02-22T20:25:34.208 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:34.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:34.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:34.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_39 2026-02-22T20:25:34.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2026-02-22T20:25:34.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:34.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:34.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:25:34.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:34.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2026-02-22T20:25:34.849 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:34.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_39 2026-02-22T20:25:34.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:34.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:34.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:25:34.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:34.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:34.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_39 2026-02-22T20:25:34.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:34.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:34.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:25:34.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2026-02-22T20:25:34.895 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2026-02-22T20:25:35.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:35.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:35.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:35.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_40 2026-02-22T20:25:35.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2026-02-22T20:25:35.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:35.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:35.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:25:35.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:35.034 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2026-02-22T20:25:36.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:36.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_40 2026-02-22T20:25:36.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:36.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:36.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:25:36.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:36.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:36.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_40 2026-02-22T20:25:36.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:36.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:36.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:25:36.045 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2026-02-22T20:25:36.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2026-02-22T20:25:36.429 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_41 2026-02-22T20:25:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2026-02-22T20:25:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:25:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2026-02-22T20:25:37.028 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_41 2026-02-22T20:25:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:25:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:37.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:37.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_41 2026-02-22T20:25:37.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:37.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:37.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:25:37.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2026-02-22T20:25:37.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2026-02-22T20:25:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_42 2026-02-22T20:25:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2026-02-22T20:25:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:25:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2026-02-22T20:25:38.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:38.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_42 2026-02-22T20:25:38.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:38.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:38.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:25:38.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:38.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_42 2026-02-22T20:25:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:25:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2026-02-22T20:25:38.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2026-02-22T20:25:38.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:38.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:38.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:38.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_43 2026-02-22T20:25:38.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2026-02-22T20:25:38.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:38.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:38.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:25:38.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:38.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2026-02-22T20:25:39.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:39.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_43 2026-02-22T20:25:39.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:39.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:39.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:25:39.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:39.050 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:39.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_43 2026-02-22T20:25:39.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:39.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:39.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:25:39.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2026-02-22T20:25:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2026-02-22T20:25:39.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_44 2026-02-22T20:25:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2026-02-22T20:25:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:25:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2026-02-22T20:25:39.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:39.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_44 2026-02-22T20:25:39.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:39.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:39.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:25:39.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:39.871 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:39.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_44 2026-02-22T20:25:39.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:39.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:39.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:25:39.872 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2026-02-22T20:25:39.903 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2026-02-22T20:25:40.046 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:40.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:40.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:40.051 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_45 2026-02-22T20:25:40.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2026-02-22T20:25:40.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:40.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:40.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:25:40.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:40.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2026-02-22T20:25:41.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_45 2026-02-22T20:25:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:25:41.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:41.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:41.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_45 2026-02-22T20:25:41.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:41.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:41.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:25:41.058 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2026-02-22T20:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2026-02-22T20:25:41.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:41.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:41.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:41.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_46 2026-02-22T20:25:41.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2026-02-22T20:25:41.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:41.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:41.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:25:41.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:41.236 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2026-02-22T20:25:42.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:42.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_46 2026-02-22T20:25:42.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:42.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:42.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:25:42.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:42.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:42.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_46 2026-02-22T20:25:42.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:25:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2026-02-22T20:25:42.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2026-02-22T20:25:42.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:42.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:42.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:42.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_47 2026-02-22T20:25:42.227 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2026-02-22T20:25:42.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:42.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:42.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:25:42.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:42.228 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2026-02-22T20:25:43.047 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_47 2026-02-22T20:25:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:25:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:43.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_47 2026-02-22T20:25:43.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:43.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:43.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:25:43.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2026-02-22T20:25:43.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2026-02-22T20:25:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:43.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:43.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:43.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_48 2026-02-22T20:25:43.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2026-02-22T20:25:43.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:43.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:43.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:25:43.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:43.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2026-02-22T20:25:44.054 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:44.073 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_48 2026-02-22T20:25:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:25:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:44.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_48 2026-02-22T20:25:44.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:44.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:44.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:25:44.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2026-02-22T20:25:44.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2026-02-22T20:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_49 2026-02-22T20:25:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2026-02-22T20:25:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:25:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2026-02-22T20:25:44.853 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:44.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_49 2026-02-22T20:25:44.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:44.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:44.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:25:44.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:44.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_49 2026-02-22T20:25:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:25:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2026-02-22T20:25:44.896 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2026-02-22T20:25:45.036 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_50 2026-02-22T20:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2026-02-22T20:25:45.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:45.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:45.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:25:45.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local snap=snap 2026-02-22T20:25:45.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2026-02-22T20:25:46.052 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing snap: 100% complete...done. 2026-02-22T20:25:46.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_50 2026-02-22T20:25:46.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:46.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:25:46.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:25:46.067 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:25:46.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_50 2026-02-22T20:25:46.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:46.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:25:46.068 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2026-02-22T20:25:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2026-02-22T20:25:46.235 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:25:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ testlog 'TEST: image deletions should propagate' 2026-02-22T20:25:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ tee -a /tmp/tmp.Xb7Kt1LMAh/rbd-mirror.test.log 2026-02-22T20:25:46.241 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ date '+%F %T' 2026-02-22T20:25:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo 2026-02-22 20:25:46 TEST: image deletions should propagate 2026-02-22T20:25:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:2026-02-22 20:25:46 TEST: image deletions should propagate 2026-02-22T20:25:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_pool_images cluster1 mirror 0 2026-02-22T20:25:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_count=0 2026-02-22T20:25:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local count 2026-02-22T20:25:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local last_count=0 2026-02-22T20:25:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ true 2026-02-22T20:25:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 40 2026-02-22T20:25:46.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in `seq 1 40` 2026-02-22T20:25:46.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 1 -ne 1 2026-02-22T20:25:46.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-22T20:25:46.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep 'images: ' 2026-02-22T20:25:46.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut '-d ' -f 2 2026-02-22T20:25:46.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: failed to query services: (13) Permission denied 2026-02-22T20:25:46.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ count=0 2026-02-22T20:25:46.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 = 0 2026-02-22T20:25:46.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:46.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2026-02-22T20:25:46.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:46.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:46.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state 2026-02-22T20:25:46.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 40 2026-02-22T20:25:46.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in `seq 1 40` 2026-02-22T20:25:46.271 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 1 -ne 1 2026-02-22T20:25:46.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-22T20:25:46.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep 'image health:' 2026-02-22T20:25:46.272 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cut '-d ' -f 3 2026-02-22T20:25:46.294 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: failed to query services: (13) Permission denied 2026-02-22T20:25:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ state=OK 2026-02-22T20:25:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test OK = ERROR 2026-02-22T20:25:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test OK = OK 2026-02-22T20:25:46.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 50 2026-02-22T20:25:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_1 2026-02-22T20:25:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2026-02-22T20:25:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:25:46.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:25:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:46.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2026-02-22T20:25:46.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:46.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.426 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:46.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:25:46.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_2 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_2 2026-02-22T20:25:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:25:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-22T20:25:46.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:46.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_2: (2) No such file or directory 2026-02-22T20:25:46.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:46.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:46.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:46.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2026-02-22T20:25:46.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:46.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:25:46.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:46.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:46.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:46.574 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:46.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_2 2026-02-22T20:25:46.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:46.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:46.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:25:46.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-22T20:25:46.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_2: (2) No such file or directory 2026-02-22T20:25:46.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:46.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:46.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:46.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:46.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:46.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_3 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_3 2026-02-22T20:25:46.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:46.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:46.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:25:46.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-22T20:25:46.603 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_3: (2) No such file or directory 2026-02-22T20:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:46.826 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:46.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2026-02-22T20:25:46.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:46.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:25:46.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:46.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_3 2026-02-22T20:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-22T20:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:46.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_3: (2) No such file or directory 2026-02-22T20:25:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_4 2026-02-22T20:25:46.953 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_4 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:25:46.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-22T20:25:46.955 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:46.974 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_4: (2) No such file or directory 2026-02-22T20:25:46.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:46.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:46.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:47.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2026-02-22T20:25:47.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_4 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:25:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-22T20:25:47.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_4: (2) No such file or directory 2026-02-22T20:25:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_5 2026-02-22T20:25:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_5 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-22T20:25:47.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_5: (2) No such file or directory 2026-02-22T20:25:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:47.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2026-02-22T20:25:47.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:25:47.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:47.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_5 2026-02-22T20:25:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:25:47.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-22T20:25:47.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_5: (2) No such file or directory 2026-02-22T20:25:47.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:47.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:47.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:47.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:47.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_6 2026-02-22T20:25:47.258 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_6 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:25:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-22T20:25:47.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_6: (2) No such file or directory 2026-02-22T20:25:47.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:47.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:47.284 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:47.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2026-02-22T20:25:47.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:25:47.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:47.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:47.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:47.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_6 2026-02-22T20:25:47.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:25:47.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-22T20:25:47.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_6: (2) No such file or directory 2026-02-22T20:25:47.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:47.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:47.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:47.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:47.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_7 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_7 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:25:47.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-22T20:25:47.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_7: (2) No such file or directory 2026-02-22T20:25:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:47.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2026-02-22T20:25:47.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:25:47.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_7 2026-02-22T20:25:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:25:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-22T20:25:47.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_7: (2) No such file or directory 2026-02-22T20:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_8 2026-02-22T20:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2026-02-22T20:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_8 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-22T20:25:47.564 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.585 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_8: (2) No such file or directory 2026-02-22T20:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2026-02-22T20:25:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:25:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:47.690 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:47.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_8 2026-02-22T20:25:47.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:25:47.691 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-22T20:25:47.692 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.713 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_8: (2) No such file or directory 2026-02-22T20:25:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_9 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_9 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:25:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-22T20:25:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.738 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_9: (2) No such file or directory 2026-02-22T20:25:47.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:47.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:47.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2026-02-22T20:25:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:25:47.843 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_9 2026-02-22T20:25:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:25:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-22T20:25:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_9: (2) No such file or directory 2026-02-22T20:25:47.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:47.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_10 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:47.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:47.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_10 2026-02-22T20:25:47.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:25:47.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-22T20:25:47.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_10: (2) No such file or directory 2026-02-22T20:25:47.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:47.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:47.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:47.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2026-02-22T20:25:47.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:47.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_10 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-22T20:25:47.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_10: (2) No such file or directory 2026-02-22T20:25:48.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:48.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:48.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:48.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:48.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_11 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_11 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_11 2026-02-22T20:25:48.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2026-02-22T20:25:48.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_11: (2) No such file or directory 2026-02-22T20:25:48.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:48.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:48.044 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2026-02-22T20:25:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_11 2026-02-22T20:25:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_11 2026-02-22T20:25:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_11 2026-02-22T20:25:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2026-02-22T20:25:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.166 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_11: (2) No such file or directory 2026-02-22T20:25:48.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:48.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:48.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_12 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_12 2026-02-22T20:25:48.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:25:48.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2026-02-22T20:25:48.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_12: (2) No such file or directory 2026-02-22T20:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2026-02-22T20:25:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_12 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_12 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_12 2026-02-22T20:25:48.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2026-02-22T20:25:48.299 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_12: (2) No such file or directory 2026-02-22T20:25:48.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:48.321 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_13 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:48.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_13 2026-02-22T20:25:48.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:25:48.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2026-02-22T20:25:48.323 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.340 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_13: (2) No such file or directory 2026-02-22T20:25:48.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:48.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:48.343 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:48.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2026-02-22T20:25:48.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_13 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_13 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_13 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2026-02-22T20:25:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_13: (2) No such file or directory 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_14 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_14 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2026-02-22T20:25:48.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_14: (2) No such file or directory 2026-02-22T20:25:48.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:48.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:48.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:48.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_14 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_14 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.593 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_14 2026-02-22T20:25:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2026-02-22T20:25:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_14: (2) No such file or directory 2026-02-22T20:25:48.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:48.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:48.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:48.616 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_15 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_15 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_15 2026-02-22T20:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2026-02-22T20:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_15: (2) No such file or directory 2026-02-22T20:25:48.640 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:48.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:48.641 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2026-02-22T20:25:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_15 2026-02-22T20:25:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_15 2026-02-22T20:25:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_15 2026-02-22T20:25:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2026-02-22T20:25:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_15: (2) No such file or directory 2026-02-22T20:25:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_16 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_16 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:25:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2026-02-22T20:25:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.788 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_16: (2) No such file or directory 2026-02-22T20:25:48.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:48.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:48.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:48.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2026-02-22T20:25:48.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_16 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_16 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_16 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2026-02-22T20:25:48.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_16: (2) No such file or directory 2026-02-22T20:25:48.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:48.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:48.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:48.916 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_17 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_17 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:48.917 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:48.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:25:48.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2026-02-22T20:25:48.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:48.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_17: (2) No such file or directory 2026-02-22T20:25:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2026-02-22T20:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_17 2026-02-22T20:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_17 2026-02-22T20:25:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_17 2026-02-22T20:25:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2026-02-22T20:25:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.060 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_17: (2) No such file or directory 2026-02-22T20:25:49.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:49.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:49.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:49.064 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_18 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_18 2026-02-22T20:25:49.065 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:25:49.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2026-02-22T20:25:49.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_18: (2) No such file or directory 2026-02-22T20:25:49.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:49.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:49.088 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2026-02-22T20:25:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_18 2026-02-22T20:25:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_18 2026-02-22T20:25:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_18 2026-02-22T20:25:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2026-02-22T20:25:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_18: (2) No such file or directory 2026-02-22T20:25:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_19 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_19 2026-02-22T20:25:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:25:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2026-02-22T20:25:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.239 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_19: (2) No such file or directory 2026-02-22T20:25:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:49.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:49.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2026-02-22T20:25:49.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_19 2026-02-22T20:25:49.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:49.344 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_19 2026-02-22T20:25:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_19 2026-02-22T20:25:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2026-02-22T20:25:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_19: (2) No such file or directory 2026-02-22T20:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_20 2026-02-22T20:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2026-02-22T20:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_20 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2026-02-22T20:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_20: (2) No such file or directory 2026-02-22T20:25:49.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:49.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:49.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:49.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_20 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_20 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_20 2026-02-22T20:25:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2026-02-22T20:25:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_20: (2) No such file or directory 2026-02-22T20:25:49.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:49.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_21 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_21 2026-02-22T20:25:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:25:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2026-02-22T20:25:49.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_21: (2) No such file or directory 2026-02-22T20:25:49.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:49.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:49.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:49.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2026-02-22T20:25:49.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_21 2026-02-22T20:25:49.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:49.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:49.642 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:49.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_21 2026-02-22T20:25:49.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_21 2026-02-22T20:25:49.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2026-02-22T20:25:49.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.662 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_21: (2) No such file or directory 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_22 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.666 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_22 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_22 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2026-02-22T20:25:49.667 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.685 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_22: (2) No such file or directory 2026-02-22T20:25:49.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:49.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:49.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:49.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2026-02-22T20:25:49.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_22 2026-02-22T20:25:49.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:49.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.790 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_22 2026-02-22T20:25:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_22 2026-02-22T20:25:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2026-02-22T20:25:49.791 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.810 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_22: (2) No such file or directory 2026-02-22T20:25:49.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:49.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:49.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:49.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:49.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_23 2026-02-22T20:25:49.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_23 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:25:49.815 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2026-02-22T20:25:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.834 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_23: (2) No such file or directory 2026-02-22T20:25:49.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:49.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2026-02-22T20:25:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_23 2026-02-22T20:25:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_23 2026-02-22T20:25:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_23 2026-02-22T20:25:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2026-02-22T20:25:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_23: (2) No such file or directory 2026-02-22T20:25:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_24 2026-02-22T20:25:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2026-02-22T20:25:49.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_24 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2026-02-22T20:25:49.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_24: (2) No such file or directory 2026-02-22T20:25:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:50.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_24 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_24 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.091 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_24 2026-02-22T20:25:50.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2026-02-22T20:25:50.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_24: (2) No such file or directory 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_25 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_25 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2026-02-22T20:25:50.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_25: (2) No such file or directory 2026-02-22T20:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2026-02-22T20:25:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_25 2026-02-22T20:25:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:50.242 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_25 2026-02-22T20:25:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_25 2026-02-22T20:25:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2026-02-22T20:25:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_25: (2) No such file or directory 2026-02-22T20:25:50.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:50.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:50.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:50.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:50.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_26 2026-02-22T20:25:50.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_26 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:25:50.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2026-02-22T20:25:50.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.287 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_26: (2) No such file or directory 2026-02-22T20:25:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:50.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2026-02-22T20:25:50.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_26 2026-02-22T20:25:50.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:50.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:50.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:50.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_26 2026-02-22T20:25:50.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_26 2026-02-22T20:25:50.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2026-02-22T20:25:50.394 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_26: (2) No such file or directory 2026-02-22T20:25:50.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:50.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:50.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:50.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:50.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_27 2026-02-22T20:25:50.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2026-02-22T20:25:50.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_27 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:25:50.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2026-02-22T20:25:50.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_27: (2) No such file or directory 2026-02-22T20:25:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_27 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_27 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_27 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2026-02-22T20:25:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_27: (2) No such file or directory 2026-02-22T20:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_28 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_28 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2026-02-22T20:25:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.588 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_28: (2) No such file or directory 2026-02-22T20:25:50.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:50.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:50.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:50.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2026-02-22T20:25:50.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_28 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_28 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_28 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2026-02-22T20:25:50.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_28: (2) No such file or directory 2026-02-22T20:25:50.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:50.719 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_29 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:50.720 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_29 2026-02-22T20:25:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:25:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2026-02-22T20:25:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_29: (2) No such file or directory 2026-02-22T20:25:50.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:50.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:50.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:50.844 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_29 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_29 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.845 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_29 2026-02-22T20:25:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2026-02-22T20:25:50.846 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_29: (2) No such file or directory 2026-02-22T20:25:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_30 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:50.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_30 2026-02-22T20:25:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_30 2026-02-22T20:25:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2026-02-22T20:25:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:50.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_30: (2) No such file or directory 2026-02-22T20:25:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:50.893 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:50.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_30 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_30 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:50.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_30 2026-02-22T20:25:50.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2026-02-22T20:25:50.996 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_30: (2) No such file or directory 2026-02-22T20:25:51.019 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_31 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_31 2026-02-22T20:25:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:25:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2026-02-22T20:25:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_31: (2) No such file or directory 2026-02-22T20:25:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2026-02-22T20:25:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_31 2026-02-22T20:25:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_31 2026-02-22T20:25:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_31 2026-02-22T20:25:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2026-02-22T20:25:51.147 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.167 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_31: (2) No such file or directory 2026-02-22T20:25:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_32 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_32 2026-02-22T20:25:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:25:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2026-02-22T20:25:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_32: (2) No such file or directory 2026-02-22T20:25:51.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:51.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:51.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:51.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2026-02-22T20:25:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_32 2026-02-22T20:25:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_32 2026-02-22T20:25:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_32 2026-02-22T20:25:51.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2026-02-22T20:25:51.298 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_32: (2) No such file or directory 2026-02-22T20:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_33 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:51.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_33 2026-02-22T20:25:51.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_33 2026-02-22T20:25:51.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2026-02-22T20:25:51.328 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.347 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_33: (2) No such file or directory 2026-02-22T20:25:51.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:51.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:51.350 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:51.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2026-02-22T20:25:51.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_33 2026-02-22T20:25:51.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:51.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:51.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_33 2026-02-22T20:25:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_33 2026-02-22T20:25:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2026-02-22T20:25:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_33: (2) No such file or directory 2026-02-22T20:25:51.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:51.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:51.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_34 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_34 2026-02-22T20:25:51.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:25:51.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2026-02-22T20:25:51.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_34: (2) No such file or directory 2026-02-22T20:25:51.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:51.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:51.505 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2026-02-22T20:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_34 2026-02-22T20:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_34 2026-02-22T20:25:51.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_34 2026-02-22T20:25:51.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2026-02-22T20:25:51.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_34: (2) No such file or directory 2026-02-22T20:25:51.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:51.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:51.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:51.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_35 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_35 2026-02-22T20:25:51.633 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:25:51.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2026-02-22T20:25:51.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_35: (2) No such file or directory 2026-02-22T20:25:51.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:51.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:51.655 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2026-02-22T20:25:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_35 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_35 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_35 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2026-02-22T20:25:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_35: (2) No such file or directory 2026-02-22T20:25:51.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:51.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.781 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_36 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:51.782 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_36 2026-02-22T20:25:51.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:25:51.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2026-02-22T20:25:51.783 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.801 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_36: (2) No such file or directory 2026-02-22T20:25:51.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:51.804 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:51.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2026-02-22T20:25:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_36 2026-02-22T20:25:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_36 2026-02-22T20:25:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_36 2026-02-22T20:25:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2026-02-22T20:25:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_36: (2) No such file or directory 2026-02-22T20:25:51.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:51.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:51.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:51.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:51.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_37 2026-02-22T20:25:51.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2026-02-22T20:25:51.935 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_37 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2026-02-22T20:25:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:51.957 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_37: (2) No such file or directory 2026-02-22T20:25:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_37 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_37 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_37 2026-02-22T20:25:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2026-02-22T20:25:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.081 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_37: (2) No such file or directory 2026-02-22T20:25:52.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:52.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:52.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:52.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:52.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_38 2026-02-22T20:25:52.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2026-02-22T20:25:52.086 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_38 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2026-02-22T20:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_38: (2) No such file or directory 2026-02-22T20:25:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:52.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2026-02-22T20:25:52.209 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_38 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_38 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_38 2026-02-22T20:25:52.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2026-02-22T20:25:52.211 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_38: (2) No such file or directory 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_39 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_39 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:25:52.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2026-02-22T20:25:52.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_39: (2) No such file or directory 2026-02-22T20:25:52.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:52.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:52.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:52.362 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2026-02-22T20:25:52.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_39 2026-02-22T20:25:52.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:52.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.363 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:52.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:52.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_39 2026-02-22T20:25:52.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_39 2026-02-22T20:25:52.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2026-02-22T20:25:52.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_39: (2) No such file or directory 2026-02-22T20:25:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:52.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_40 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_40 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:25:52.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2026-02-22T20:25:52.388 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_40: (2) No such file or directory 2026-02-22T20:25:52.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:52.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:52.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:52.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2026-02-22T20:25:52.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_40 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_40 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_40 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2026-02-22T20:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_40: (2) No such file or directory 2026-02-22T20:25:52.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:52.537 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_41 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:52.538 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_41 2026-02-22T20:25:52.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:25:52.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2026-02-22T20:25:52.539 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_41: (2) No such file or directory 2026-02-22T20:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:52.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2026-02-22T20:25:52.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_41 2026-02-22T20:25:52.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:52.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:52.663 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:52.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_41 2026-02-22T20:25:52.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_41 2026-02-22T20:25:52.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2026-02-22T20:25:52.664 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.683 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_41: (2) No such file or directory 2026-02-22T20:25:52.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:52.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:52.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_42 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_42 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:25:52.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2026-02-22T20:25:52.689 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_42: (2) No such file or directory 2026-02-22T20:25:52.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:52.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:52.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:52.812 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2026-02-22T20:25:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_42 2026-02-22T20:25:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_42 2026-02-22T20:25:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_42 2026-02-22T20:25:52.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2026-02-22T20:25:52.814 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_42: (2) No such file or directory 2026-02-22T20:25:52.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:52.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:52.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_43 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_43 2026-02-22T20:25:52.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:25:52.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2026-02-22T20:25:52.838 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.859 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_43: (2) No such file or directory 2026-02-22T20:25:52.862 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:52.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:52.863 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:52.964 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2026-02-22T20:25:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_43 2026-02-22T20:25:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_43 2026-02-22T20:25:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_43 2026-02-22T20:25:52.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2026-02-22T20:25:52.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:52.985 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_43: (2) No such file or directory 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_44 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:52.990 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_44 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2026-02-22T20:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_44: (2) No such file or directory 2026-02-22T20:25:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:53.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:53.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_44 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_44 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_44 2026-02-22T20:25:53.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2026-02-22T20:25:53.116 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.137 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_44: (2) No such file or directory 2026-02-22T20:25:53.141 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:53.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:53.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:53.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:53.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_45 2026-02-22T20:25:53.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2026-02-22T20:25:53.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.142 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_45 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2026-02-22T20:25:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_45: (2) No such file or directory 2026-02-22T20:25:53.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:53.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:53.165 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:53.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2026-02-22T20:25:53.266 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_45 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_45 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_45 2026-02-22T20:25:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2026-02-22T20:25:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_45: (2) No such file or directory 2026-02-22T20:25:53.291 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_46 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:25:53.292 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_46 2026-02-22T20:25:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:25:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2026-02-22T20:25:53.293 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.311 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_46: (2) No such file or directory 2026-02-22T20:25:53.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:53.314 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:53.315 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2026-02-22T20:25:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_46 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_46 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_46 2026-02-22T20:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2026-02-22T20:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_46: (2) No such file or directory 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_47 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_47 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2026-02-22T20:25:53.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.460 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_47: (2) No such file or directory 2026-02-22T20:25:53.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:53.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:53.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:53.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2026-02-22T20:25:53.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_47 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_47 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_47 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2026-02-22T20:25:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.586 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_47: (2) No such file or directory 2026-02-22T20:25:53.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:53.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:53.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:53.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:53.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_48 2026-02-22T20:25:53.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_48 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.591 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:25:53.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2026-02-22T20:25:53.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.609 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_48: (2) No such file or directory 2026-02-22T20:25:53.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:53.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:53.612 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2026-02-22T20:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_48 2026-02-22T20:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:53.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_48 2026-02-22T20:25:53.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.715 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_48 2026-02-22T20:25:53.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2026-02-22T20:25:53.716 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_48: (2) No such file or directory 2026-02-22T20:25:53.740 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_49 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_49 2026-02-22T20:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2026-02-22T20:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.760 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_49: (2) No such file or directory 2026-02-22T20:25:53.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:53.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:53.764 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:53.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2026-02-22T20:25:53.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_49 2026-02-22T20:25:53.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:53.867 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:53.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:53.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_49 2026-02-22T20:25:53.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.868 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_49 2026-02-22T20:25:53.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2026-02-22T20:25:53.869 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_49: (2) No such file or directory 2026-02-22T20:25:53.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-22T20:25:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_50 2026-02-22T20:25:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2026-02-22T20:25:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:53.890 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_50 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_50 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2026-02-22T20:25:53.891 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:53.909 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_50: (2) No such file or directory 2026-02-22T20:25:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:25:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:25:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_50 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_50 2026-02-22T20:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:25:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:25:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_50 2026-02-22T20:25:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2026-02-22T20:25:54.014 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:25:54.037 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_50: (2) No such file or directory 2026-02-22T20:25:54.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:25:54.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:25:54.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:25:54.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:25:54.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ testlog 'TEST: delete images during bootstrap' 2026-02-22T20:25:54.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ tee -a /tmp/tmp.Xb7Kt1LMAh/rbd-mirror.test.log 2026-02-22T20:25:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ date '+%F %T' 2026-02-22T20:25:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo 2026-02-22 20:25:54 TEST: delete images during bootstrap 2026-02-22T20:25:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:2026-02-22 20:25:54 TEST: delete images during bootstrap 2026-02-22T20:25:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_pool_mirror_mode cluster1 mirror image 2026-02-22T20:25:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=image 2026-02-22T20:25:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2026-02-22T20:25:54.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:note: changing mirroring mode from pool to image 2026-02-22T20:25:54.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_pool_mirror_mode cluster2 mirror image 2026-02-22T20:25:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=image 2026-02-22T20:25:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2026-02-22T20:25:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:note: changing mirroring mode from pool to image 2026-02-22T20:25:54.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ start_mirror cluster1 2026-02-22T20:25:54.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:25:54.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance 2026-02-22T20:25:54.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set_cluster_instance cluster1 cluster instance 2026-02-22T20:25:54.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local val=cluster1 2026-02-22T20:25:54.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster_var_name=cluster 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local instance_var_name=instance 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cluster=cluster1 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=cluster1 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' cluster1 = cluster1 ']' 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ instance=0 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval cluster=cluster1 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ cluster=cluster1 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ eval instance=0 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ instance=0 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T20:25:54.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=test 2026-02-22T20:25:54.112 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 1 10 2026-02-22T20:25:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:25:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_1 2026-02-22T20:25:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_1 512M 2026-02-22T20:25:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:25:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:54.113 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:25:54.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:25:54.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:25:54.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:25:54.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:25:54.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:25:54.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:25:54.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_1 2026-02-22T20:25:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_1 2026-02-22T20:25:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:25:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:25:54.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 journal 2026-02-22T20:25:54.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:25:54.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2026-02-22T20:25:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_1': 2026-02-22T20:25:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:25:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:25:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:25:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 37c667bb5b4 2026-02-22T20:25:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.37c667bb5b4 2026-02-22T20:25:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:25:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:25:54 2026 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:25:54 2026 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:25:54 2026 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 37c667bb5b4 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: df1a8a1f-1ce4-4cc4-9474-053777bd5a48 2026-02-22T20:25:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:25:54.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_1 2026-02-22T20:25:54.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:25:54.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:25:54.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:25:54.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:25:54.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=7 2026-02-22T20:25:54.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:25:54.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:25:55.215 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 54970368 bytes 2026-02-22T20:25:55.600 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 93375488 bytes 2026-02-22T20:25:55.837 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 104095744 bytes 2026-02-22T20:25:57.835 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 221635584 bytes 2026-02-22T20:25:57.896 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8227725571364604) > weight (0.25) 2026-02-22T20:25:57.896 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7390192587446756) > weight (0.25) 2026-02-22T20:25:57.896 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7708896552368267) > weight (0.25) 2026-02-22T20:25:57.897 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2026-02-22T20:25:57.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2026-02-22T20:25:57.897 INFO:tasks.rbd_mirror_thrash:waiting for 2.139233701692765 secs before reviving daemons 2026-02-22T20:25:57.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:25:57.892+0000 7f98ca378640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 20031) UID: 1000 2026-02-22T20:25:59.022 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 276161536 bytes 2026-02-22T20:26:00.038 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2026-02-22T20:26:00.038 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:26:00.038 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2026-02-22T20:26:00.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2026-02-22T20:26:00.038 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2026-02-22T20:26:00.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2026-02-22T20:26:00.041 INFO:tasks.rbd_mirror_thrash:waiting for 105.41677813275365 secs before thrashing 2026-02-22T20:26:00.283 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 368500736 bytes 2026-02-22T20:26:01.074 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 392860672 bytes 2026-02-22T20:26:01.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:26:01.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:26:01.230 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:01.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:01.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:01.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:01.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:26:01.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:26:01.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:01.257 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:26:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 present '' 2026-02-22T20:26:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:26:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:01.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:01.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:01.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:01.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:01.360 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:01.380 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:26:01.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:26:01.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:01.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = deleted 2026-02-22T20:26:01.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:01.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:26:02.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 present '' 2026-02-22T20:26:02.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:02.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:02.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:02.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:26:02.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:02.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:02.385 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:02.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:02.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:02.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:02.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:02.386 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:02.387 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:02.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:26:02.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:26:02.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:02.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = deleted 2026-02-22T20:26:02.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:02.410 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 2 2026-02-22T20:26:04.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 present '' 2026-02-22T20:26:04.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:04.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:04.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:04.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:26:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:26:04.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:26:04.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:04.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = deleted 2026-02-22T20:26:04.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:04.438 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 4 2026-02-22T20:26:08.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 present '' 2026-02-22T20:26:08.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:08.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:08.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:08.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:26:08.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:26:08.467 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:08.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = deleted 2026-02-22T20:26:08.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:08.468 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 8 2026-02-22T20:26:16.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 present '' 2026-02-22T20:26:16.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:16.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:16.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:16.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:26:16.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:16.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:16.469 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:16.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:16.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:16.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:16.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:16.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:16.470 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:16.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:26:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:26:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = deleted 2026-02-22T20:26:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:16.496 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 8 2026-02-22T20:26:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 present '' 2026-02-22T20:26:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:26:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:24.502 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:26:24.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:26:24.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:24.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = deleted 2026-02-22T20:26:24.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:24.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 8 2026-02-22T20:26:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 present '' 2026-02-22T20:26:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:32.561 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a5f78e0b65 2026-02-22T20:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a5f78e0b65 2026-02-22T20:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z '' -o '' = 12a5f78e0b65 2026-02-22T20:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_1 2026-02-22T20:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:32.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2026-02-22T20:26:36.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:26:36.309+0000 7f6cab6fe640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56185cee1140 finish: resending after timeout 2026-02-22T20:26:41.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:26:41.309+0000 7f6caaefd640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56185cee1140 finish: resending after timeout 2026-02-22T20:26:46.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:26:46.309+0000 7f6caaefd640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56185cee1140 finish: resending after timeout 2026-02-22T20:26:47.748 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:47.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a5f78e0b65 2026-02-22T20:26:47.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:47.786 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:26:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 deleted 12a5f78e0b65 2026-02-22T20:26:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a5f78e0b65 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:47.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a5f78e0b65 2026-02-22T20:26:47.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a5f78e0b65 2026-02-22T20:26:47.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a5f78e0b65 -o 12a5f78e0b65 = 12a5f78e0b65 2026-02-22T20:26:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:26:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = present 2026-02-22T20:26:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 1 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_1 deleted 12a5f78e0b65 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a5f78e0b65 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_1 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_1 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-22T20:26:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:48.946 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-22T20:26:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:26:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:26:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:26:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_1 2026-02-22T20:26:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:26:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2026-02-22T20:26:49.437 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-22T20:26:49.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-22T20:26:49.438 INFO:tasks.rbd_mirror_thrash:waiting for 7.2936247297897205 secs before reviving daemons 2026-02-22T20:26:49.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:26:49.433+0000 7fe6cf145640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 93889) UID: 1000 2026-02-22T20:26:49.879 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-22T20:26:49.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_1 2026-02-22T20:26:49.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:26:49.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:49.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:49.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:26:49.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:26:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_1 2026-02-22T20:26:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:26:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_1 2026-02-22T20:26:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2026-02-22T20:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2026-02-22T20:26:50.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_2 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_2 512M 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:26:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:26:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:26:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_2 2026-02-22T20:26:50.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_2 2026-02-22T20:26:50.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:26:50.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:50.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:26:50.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:26:50.114 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 journal 2026-02-22T20:26:50.145 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:26:50.151 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2026-02-22T20:26:50.180 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_2': 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 37e37f21d342 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.37e37f21d342 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:26:50 2026 2026-02-22T20:26:50.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:26:50 2026 2026-02-22T20:26:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:26:50 2026 2026-02-22T20:26:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 37e37f21d342 2026-02-22T20:26:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:26:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:26:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 024e7f0f-b994-437c-9ad4-5c12e6c9df25 2026-02-22T20:26:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:26:50.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_2 2026-02-22T20:26:50.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:26:50.184 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:50.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:26:50.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:26:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=8 2026-02-22T20:26:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:26:50.187 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:26:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 46512128 bytes 2026-02-22T20:26:51.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 76242944 bytes 2026-02-22T20:26:51.671 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 102033408 bytes 2026-02-22T20:26:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 147310592 bytes 2026-02-22T20:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 181632000 bytes 2026-02-22T20:26:54.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 245786624 bytes 2026-02-22T20:26:55.701 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 268748800 bytes 2026-02-22T20:26:56.732 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-22T20:26:56.733 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:26:56.733 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-22T20:26:56.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-22T20:26:56.734 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-22T20:26:56.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-22T20:26:56.736 INFO:tasks.rbd_mirror_thrash:waiting for 76.8191391062937 secs before thrashing 2026-02-22T20:26:56.794 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 315824128 bytes 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:26:58.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_2 2026-02-22T20:26:58.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:58.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:58.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:26:58.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-22T20:26:58.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:58.226 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:26:58.212+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4455bd40 2026-02-22T20:26:58.226 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:26:58.212+0000 7f5e092a6640 -1 reset not still connected to 0x55ac64769ba0 2026-02-22T20:26:58.226 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:26:58.212+0000 7f5e092a6640 -1 reset not still connected to 0x55ac684fbee0 2026-02-22T20:26:58.226 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:26:58.212+0000 7f5e092a6640 -1 reset not still connected to 0x55ac4455bd40 2026-02-22T20:26:58.226 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:26:58.212+0000 7f5e092a6640 -1 reset not still connected to 0x55ac64769ba0 2026-02-22T20:26:58.226 INFO:tasks.ceph.cluster1.osd.2.trial042.stderr:2026-02-22T20:26:58.212+0000 7f5e092a6640 -1 reset not still connected to 0x55ac684fbee0 2026-02-22T20:26:58.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a5e31c2483 2026-02-22T20:26:58.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:26:58.229 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:26:58.330 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_2 present 12a5e31c2483 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a5e31c2483 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_2 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-22T20:26:58.331 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:26:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a5e31c2483 2026-02-22T20:26:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a5e31c2483 2026-02-22T20:26:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a5e31c2483 -o 12a5e31c2483 = 12a5e31c2483 2026-02-22T20:26:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:26:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:26:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:26:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_2 2026-02-22T20:26:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:26:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:26:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:26:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2026-02-22T20:26:58.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:26:58.449+0000 7f7076f15640 -1 rbd::mirror::ImageMap: 0x55c9284fd0e0 handle_peer_ack_remove: failed to remove global_image_id=024e7f0f-b994-437c-9ad4-5c12e6c9df25 2026-02-22T20:27:28.997 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:27:29.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2026-02-22T20:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:27:29.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:27:29.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_2 2026-02-22T20:27:29.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:27:29.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:27:29.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:27:29.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-22T20:27:29.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:27:29.029 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_2: (2) No such file or directory 2026-02-22T20:27:29.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:27:29.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:27:29.033 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:27:29.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2026-02-22T20:27:29.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:27:29.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:29.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:27:29.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:27:29.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:27:29.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:27:29.135 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:27:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_2 2026-02-22T20:27:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:27:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:27:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_2 2026-02-22T20:27:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-22T20:27:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:27:29.158 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_2: (2) No such file or directory 2026-02-22T20:27:29.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:27:29.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:27:29.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:27:29.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:27:29.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_2 2026-02-22T20:27:29.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:27:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:27:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2026-02-22T20:27:29.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_2 2026-02-22T20:27:29.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:27:29.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:29.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:27:29.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:27:29.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:27:29.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_2 2026-02-22T20:27:29.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:27:29.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:29.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_2 2026-02-22T20:27:29.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2026-02-22T20:27:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2026-02-22T20:27:29.377 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:27:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:27:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:27:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_3 2026-02-22T20:27:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_3 512M 2026-02-22T20:27:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:27:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:27:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:27:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:27:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:27:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:27:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:27:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:27:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:27:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_3 2026-02-22T20:27:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_3 2026-02-22T20:27:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:27:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:27:29.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:27:29.420 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 journal 2026-02-22T20:27:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2026-02-22T20:27:29.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_3': 2026-02-22T20:27:29.477 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 38059477098 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.38059477098 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:27:29 2026 2026-02-22T20:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:27:29 2026 2026-02-22T20:27:29.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:27:29 2026 2026-02-22T20:27:29.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 38059477098 2026-02-22T20:27:29.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:27:29.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:27:29.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 81ae31fe-c205-49fd-9bf5-8df53d615832 2026-02-22T20:27:29.479 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:27:29.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_3 2026-02-22T20:27:29.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:27:29.480 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:29.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:27:29.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=6 2026-02-22T20:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:27:30.848 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 48843776 bytes 2026-02-22T20:27:32.012 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 134558720 bytes 2026-02-22T20:27:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 173992960 bytes 2026-02-22T20:27:33.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 174607360 bytes 2026-02-22T20:27:34.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 208245760 bytes 2026-02-22T20:27:34.570 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 223562752 bytes 2026-02-22T20:27:34.770 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 232257536 bytes 2026-02-22T20:27:35.491 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:27:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_3 2026-02-22T20:27:35.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:27:35.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:27:35.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:27:35.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-22T20:27:35.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:27:35.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a5db655970 2026-02-22T20:27:35.533 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:27:35.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:27:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_3 present 12a5db655970 2026-02-22T20:27:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:27:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a5db655970 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_3 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-22T20:27:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:27:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a5db655970 2026-02-22T20:27:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a5db655970 2026-02-22T20:27:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a5db655970 -o 12a5db655970 = 12a5db655970 2026-02-22T20:27:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:27:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:27:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:27:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_3 2026-02-22T20:27:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:27:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:27:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:27:35.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2026-02-22T20:27:45.458 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6980906720212993) > weight (0.25) 2026-02-22T20:27:45.458 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.44172609139619956) > weight (0.25) 2026-02-22T20:27:45.458 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.730745649023012) > weight (0.25) 2026-02-22T20:27:45.458 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6234656277513692) > weight (0.25) 2026-02-22T20:27:45.458 INFO:tasks.rbd_mirror_thrash:waiting for 107.14651499217013 secs before thrashing 2026-02-22T20:28:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:28:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2026-02-22T20:28:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:28:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:28:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:28:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:28:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:28:05.975 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:28:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_3 2026-02-22T20:28:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:28:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:28:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:28:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-22T20:28:05.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:28:05.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_3: (2) No such file or directory 2026-02-22T20:28:06.002 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:28:06.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:28:06.003 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:28:06.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2026-02-22T20:28:06.103 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_3 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_3 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-22T20:28:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:28:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_3: (2) No such file or directory 2026-02-22T20:28:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:28:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:28:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:28:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:28:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_3 2026-02-22T20:28:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:28:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_3 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_3 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:06.161 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_3 2026-02-22T20:28:06.162 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2026-02-22T20:28:06.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2026-02-22T20:28:06.336 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_4 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_4 512M 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:28:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:28:06.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:28:06.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:28:06.342 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_4 2026-02-22T20:28:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_4 2026-02-22T20:28:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:28:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 journal 2026-02-22T20:28:06.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:28:06.418 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2026-02-22T20:28:06.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_4': 2026-02-22T20:28:06.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:28:06.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:28:06.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:28:06.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 381d3da1f54c 2026-02-22T20:28:06.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.381d3da1f54c 2026-02-22T20:28:06.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:28:06 2026 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:28:06 2026 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:28:06 2026 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 381d3da1f54c 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: c0d0ec5b-1dad-4e57-9704-2c880afa074c 2026-02-22T20:28:06.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:28:06.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_4 2026-02-22T20:28:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:06.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:28:06.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=13 2026-02-22T20:28:06.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:28:06.450 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:28:08.021 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 62894080 bytes 2026-02-22T20:28:08.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 143303680 bytes 2026-02-22T20:28:10.148 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 181206016 bytes 2026-02-22T20:28:11.080 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 226591744 bytes 2026-02-22T20:28:11.449 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 238321664 bytes 2026-02-22T20:28:12.378 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 300775424 bytes 2026-02-22T20:28:12.882 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 328571904 bytes 2026-02-22T20:28:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 369813504 bytes 2026-02-22T20:28:13.556 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.47073336016096523) > weight (0.25) 2026-02-22T20:28:13.556 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.48880027840022877) > weight (0.25) 2026-02-22T20:28:13.556 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3092802831781236) > weight (0.25) 2026-02-22T20:28:13.556 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2026-02-22T20:28:13.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2026-02-22T20:28:13.557 INFO:tasks.rbd_mirror_thrash:waiting for 6.304975958537574 secs before reviving daemons 2026-02-22T20:28:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:28:13.553+0000 7f76aed7e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 99469) UID: 1000 2026-02-22T20:28:14.949 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 418068480 bytes 2026-02-22T20:28:15.833 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 493273088 bytes 2026-02-22T20:28:16.310 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 518522880 bytes 2026-02-22T20:28:16.523 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 527412224 bytes 2026-02-22T20:28:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 537627648 bytes 2026-02-22T20:28:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 562937856 bytes 2026-02-22T20:28:18.076 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1500 IOs, 578419712 bytes 2026-02-22T20:28:19.192 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1600 IOs, 634509312 bytes 2026-02-22T20:28:19.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:28:19.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:28:19.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:28:19.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:28:19.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2026-02-22T20:28:19.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:28:19.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:19.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:19.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:28:19.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:28:19.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:28:19.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:28:19.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_4 2026-02-22T20:28:19.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:28:19.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:28:19.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:28:19.465 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:28:19.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-22T20:28:19.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a54be5bbd7 2026-02-22T20:28:19.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:28:19.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_4 present 12a54be5bbd7 2026-02-22T20:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a54be5bbd7 2026-02-22T20:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_4 2026-02-22T20:28:19.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:28:19.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:28:19.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:28:19.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-22T20:28:19.596 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:28:19.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a54be5bbd7 2026-02-22T20:28:19.627 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a54be5bbd7 2026-02-22T20:28:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a54be5bbd7 -o 12a54be5bbd7 = 12a54be5bbd7 2026-02-22T20:28:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:28:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:28:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:28:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_4 2026-02-22T20:28:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2026-02-22T20:28:19.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:28:19.725+0000 7f7076714640 -1 rbd::mirror::ImageMap: 0x55c9284fd0e0 handle_peer_ack_remove: failed to remove global_image_id=c0d0ec5b-1dad-4e57-9704-2c880afa074c 2026-02-22T20:28:19.863 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2026-02-22T20:28:19.863 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:28:19.864 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2026-02-22T20:28:19.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2026-02-22T20:28:19.864 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2026-02-22T20:28:19.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2026-02-22T20:28:19.866 INFO:tasks.rbd_mirror_thrash:waiting for 68.12488201003012 secs before thrashing 2026-02-22T20:28:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:28:50.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2026-02-22T20:28:50.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_4 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:28:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:28:50.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-22T20:28:50.085 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:28:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_4: (2) No such file or directory 2026-02-22T20:28:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:28:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:28:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:28:50.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2026-02-22T20:28:50.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:28:50.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:50.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:50.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:28:50.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:28:50.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:28:50.213 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:28:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_4 2026-02-22T20:28:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:28:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:28:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_4 2026-02-22T20:28:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-22T20:28:50.214 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_4: (2) No such file or directory 2026-02-22T20:28:50.237 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:28:50.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:28:50.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:28:50.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:28:50.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_4 2026-02-22T20:28:50.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:50.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:50.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:50.238 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2026-02-22T20:28:50.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_4 2026-02-22T20:28:50.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:50.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:50.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:50.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:28:50.269 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:28:50.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_4 2026-02-22T20:28:50.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:50.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:50.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_4 2026-02-22T20:28:50.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2026-02-22T20:28:50.301 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2026-02-22T20:28:50.448 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_5 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_5 512M 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:28:50.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_5 2026-02-22T20:28:50.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_5 2026-02-22T20:28:50.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:50.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:50.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:28:50.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:28:50.493 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 journal 2026-02-22T20:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:28:50.530 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_5': 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 383eb768c371 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.383eb768c371 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:28:50 2026 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:28:50 2026 2026-02-22T20:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:28:50 2026 2026-02-22T20:28:50.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 383eb768c371 2026-02-22T20:28:50.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:28:50.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:28:50.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 71660584-2a4f-4894-b714-2effc35ed581 2026-02-22T20:28:50.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:28:50.562 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_5 2026-02-22T20:28:50.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:28:50.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:28:50.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:28:50.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:28:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T20:28:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:28:50.565 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:28:51.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 35771392 bytes 2026-02-22T20:28:51.769 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 68931584 bytes 2026-02-22T20:28:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 103763968 bytes 2026-02-22T20:28:53.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 166041600 bytes 2026-02-22T20:28:54.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 182679552 bytes 2026-02-22T20:28:55.089 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 221470720 bytes 2026-02-22T20:28:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 237821952 bytes 2026-02-22T20:28:57.254 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 303666176 bytes 2026-02-22T20:28:58.492 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 332807168 bytes 2026-02-22T20:28:58.921 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 369223680 bytes 2026-02-22T20:28:59.635 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 399855616 bytes 2026-02-22T20:29:00.327 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 423474176 bytes 2026-02-22T20:29:01.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 520819712 bytes 2026-02-22T20:29:02.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:29:02.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_5 2026-02-22T20:29:02.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:29:02.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:29:02.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:29:02.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-22T20:29:02.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:29:02.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a535f2bea7 2026-02-22T20:29:02.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:29:02.608 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:29:02.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_5 present 12a535f2bea7 2026-02-22T20:29:02.710 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a535f2bea7 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_5 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:29:02.711 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-22T20:29:02.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:29:02.742 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a535f2bea7 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a535f2bea7 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a535f2bea7 -o 12a535f2bea7 = 12a535f2bea7 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_5 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:29:02.743 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2026-02-22T20:29:27.991 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.32064008985254755) > weight (0.25) 2026-02-22T20:29:27.992 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2026-02-22T20:29:27.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2026-02-22T20:29:27.992 INFO:tasks.rbd_mirror_thrash:waiting for 8.59318394354439 secs before reviving daemons 2026-02-22T20:29:27.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:29:27.985+0000 7f7077716640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 112446) UID: 1000 2026-02-22T20:29:32.062 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2026-02-22T20:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_5 2026-02-22T20:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-22T20:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:29:32.096 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_5: (2) No such file or directory 2026-02-22T20:29:32.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:29:32.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:29:32.099 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:29:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2026-02-22T20:29:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:29:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:29:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:29:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:29:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:29:32.200 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:29:32.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_5 2026-02-22T20:29:32.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:29:32.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:29:32.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_5 2026-02-22T20:29:32.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-22T20:29:32.201 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:29:32.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_5: (2) No such file or directory 2026-02-22T20:29:32.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:29:32.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:29:32.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:29:32.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:29:32.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_5 2026-02-22T20:29:32.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:29:32.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:32.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:29:32.225 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2026-02-22T20:29:32.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_5 2026-02-22T20:29:32.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:29:32.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:32.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:29:32.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:29:32.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:29:32.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_5 2026-02-22T20:29:32.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:29:32.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:32.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_5 2026-02-22T20:29:32.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2026-02-22T20:29:32.296 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2026-02-22T20:29:32.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:29:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:29:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:29:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_6 2026-02-22T20:29:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_6 512M 2026-02-22T20:29:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:29:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:29:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:29:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:29:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:29:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:29:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:29:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:29:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:29:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_6 2026-02-22T20:29:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_6 2026-02-22T20:29:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:29:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:29:32.482 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:29:32.483 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 journal 2026-02-22T20:29:32.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:29:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_6': 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 38558ca22916 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.38558ca22916 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:29:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:29:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:29:32 2026 2026-02-22T20:29:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:29:32 2026 2026-02-22T20:29:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:29:32 2026 2026-02-22T20:29:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 38558ca22916 2026-02-22T20:29:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:29:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:29:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: c5ca5b9a-0b40-471a-98af-1dc271816521 2026-02-22T20:29:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:29:32.545 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_6 2026-02-22T20:29:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:29:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:29:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:29:32.547 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=12 2026-02-22T20:29:32.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:29:32.548 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:29:32.605 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4020313399158094) > weight (0.25) 2026-02-22T20:29:32.606 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-22T20:29:32.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-22T20:29:32.606 INFO:tasks.rbd_mirror_thrash:waiting for 6.375115677151399 secs before reviving daemons 2026-02-22T20:29:32.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:29:32.601+0000 7f6cabeff640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 107691) UID: 1000 2026-02-22T20:29:32.840 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 10725376 bytes 2026-02-22T20:29:33.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 40497152 bytes 2026-02-22T20:29:35.892 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 135731200 bytes 2026-02-22T20:29:36.586 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2026-02-22T20:29:36.586 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:29:36.587 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2026-02-22T20:29:36.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-22T20:29:36.587 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-22T20:29:36.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-22T20:29:36.589 INFO:tasks.rbd_mirror_thrash:waiting for 74.36294443860085 secs before thrashing 2026-02-22T20:29:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 232700928 bytes 2026-02-22T20:29:38.982 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-22T20:29:38.983 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:29:38.983 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-22T20:29:38.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-22T20:29:38.983 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-22T20:29:38.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-22T20:29:38.985 INFO:tasks.rbd_mirror_thrash:waiting for 83.54293455440441 secs before thrashing 2026-02-22T20:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 268880896 bytes 2026-02-22T20:29:40.677 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 359657472 bytes 2026-02-22T20:29:41.486 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 407289856 bytes 2026-02-22T20:29:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 417773568 bytes 2026-02-22T20:29:43.084 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 459699200 bytes 2026-02-22T20:29:43.659 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 512744448 bytes 2026-02-22T20:29:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:29:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:29:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:29:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:29:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2026-02-22T20:29:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_6 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-22T20:29:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:29:44.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a576d1643d 2026-02-22T20:29:44.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:29:44.592 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:29:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_6 present 12a576d1643d 2026-02-22T20:29:44.693 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a576d1643d 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_6 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:29:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:29:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-22T20:29:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a576d1643d 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a576d1643d 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a576d1643d -o 12a576d1643d = 12a576d1643d 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_6 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:29:44.725 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2026-02-22T20:29:44.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:29:44.817+0000 7f9f67a1f640 -1 rbd::mirror::ImageMap: 0x56251a8a0c60 handle_peer_ack_remove: failed to remove global_image_id=c5ca5b9a-0b40-471a-98af-1dc271816521 2026-02-22T20:30:15.083 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_6 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:30:15.092 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:30:15.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-22T20:30:15.093 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:30:15.117 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_6: (2) No such file or directory 2026-02-22T20:30:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:30:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:30:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:30:15.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2026-02-22T20:30:15.222 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_6 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_6 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-22T20:30:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:30:15.245 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_6: (2) No such file or directory 2026-02-22T20:30:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:30:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:30:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:30:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:30:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_6 2026-02-22T20:30:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:15.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:30:15.250 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2026-02-22T20:30:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_6 2026-02-22T20:30:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:30:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:30:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:30:15.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_6 2026-02-22T20:30:15.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:15.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:15.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_6 2026-02-22T20:30:15.282 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2026-02-22T20:30:15.313 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2026-02-22T20:30:15.466 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:30:15.471 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_7 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_7 512M 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:30:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:30:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:30:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:30:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:30:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_7 2026-02-22T20:30:15.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_7 2026-02-22T20:30:15.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:15.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:15.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:15.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:30:15.510 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 journal 2026-02-22T20:30:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:30:15.546 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_7': 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 387910d9c421 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.387910d9c421 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:30:15 2026 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:30:15 2026 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:30:15 2026 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 387910d9c421 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 6bc9a081-17a5-43f7-9874-3ec0a380f9cf 2026-02-22T20:30:15.576 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:30:15.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_7 2026-02-22T20:30:15.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:15.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:15.577 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:15.578 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:30:15.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=7 2026-02-22T20:30:15.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:30:15.579 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:30:16.734 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 56572928 bytes 2026-02-22T20:30:18.940 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 139505664 bytes 2026-02-22T20:30:19.149 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 172515328 bytes 2026-02-22T20:30:20.071 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 203290624 bytes 2026-02-22T20:30:21.643 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 329486336 bytes 2026-02-22T20:30:22.441 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 358057984 bytes 2026-02-22T20:30:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:30:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:30:22.589 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_7 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-22T20:30:22.590 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:30:22.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a51e020305 2026-02-22T20:30:22.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:30:22.624 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:30:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_7 present 12a51e020305 2026-02-22T20:30:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:30:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:30:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a51e020305 2026-02-22T20:30:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:30:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:30:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_7 2026-02-22T20:30:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:30:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:30:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:30:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-22T20:30:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:30:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a51e020305 2026-02-22T20:30:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a51e020305 2026-02-22T20:30:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a51e020305 -o 12a51e020305 = 12a51e020305 2026-02-22T20:30:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:30:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:30:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:30:22.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_7 2026-02-22T20:30:22.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:22.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:22.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:22.759 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2026-02-22T20:30:50.953 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8752601573856931) > weight (0.25) 2026-02-22T20:30:50.954 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2026-02-22T20:30:50.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2026-02-22T20:30:50.954 INFO:tasks.rbd_mirror_thrash:waiting for 6.401902542285429 secs before reviving daemons 2026-02-22T20:30:50.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:30:50.949+0000 7f71bcbd3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 129657) UID: 1000 2026-02-22T20:30:53.251 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:30:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2026-02-22T20:30:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:30:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:53.260 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_7 2026-02-22T20:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:30:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:30:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:30:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-22T20:30:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:30:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_7: (2) No such file or directory 2026-02-22T20:30:53.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:30:53.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:30:53.288 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:30:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2026-02-22T20:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:30:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_7 2026-02-22T20:30:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:30:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:30:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_7 2026-02-22T20:30:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-22T20:30:53.392 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:30:53.411 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_7: (2) No such file or directory 2026-02-22T20:30:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:30:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:30:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:30:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:30:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_7 2026-02-22T20:30:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2026-02-22T20:30:53.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_7 2026-02-22T20:30:53.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:53.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:53.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:53.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:30:53.446 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:30:53.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_7 2026-02-22T20:30:53.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:53.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:53.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_7 2026-02-22T20:30:53.447 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2026-02-22T20:30:53.484 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2026-02-22T20:30:53.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_8 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_8 512M 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_8 2026-02-22T20:30:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_8 2026-02-22T20:30:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:30:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:30:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 journal 2026-02-22T20:30:53.704 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:30:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_8': 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 388ce9d7b4d8 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.388ce9d7b4d8 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:30:53 2026 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:30:53 2026 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:30:53 2026 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 388ce9d7b4d8 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: b8aa5642-72f5-4ed6-9041-db6447ecd1ae 2026-02-22T20:30:53.736 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:30:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_8 2026-02-22T20:30:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:30:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:30:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:30:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:30:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=10 2026-02-22T20:30:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:30:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:30:54.277 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 21646336 bytes 2026-02-22T20:30:54.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 25289728 bytes 2026-02-22T20:30:54.478 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 25891840 bytes 2026-02-22T20:30:54.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 69870592 bytes 2026-02-22T20:30:56.947 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 132629504 bytes 2026-02-22T20:30:57.357 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2026-02-22T20:30:57.357 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:30:57.357 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2026-02-22T20:30:57.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-22T20:30:57.357 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-22T20:30:57.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-22T20:30:57.359 INFO:tasks.rbd_mirror_thrash:waiting for 71.12475982139237 secs before thrashing 2026-02-22T20:30:57.474 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 179749888 bytes 2026-02-22T20:30:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:700 IOs, 216396800 bytes 2026-02-22T20:30:58.567 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:800 IOs, 269502464 bytes 2026-02-22T20:30:59.967 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:900 IOs, 305252352 bytes 2026-02-22T20:31:00.969 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1000 IOs, 375796736 bytes 2026-02-22T20:31:01.925 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1100 IOs, 407012352 bytes 2026-02-22T20:31:02.529 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2026-02-22T20:31:02.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2026-02-22T20:31:02.530 INFO:tasks.rbd_mirror_thrash:waiting for 3.054835639358271 secs before reviving daemons 2026-02-22T20:31:02.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:31:02.525+0000 7fe2eec76640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 111994) UID: 1000 2026-02-22T20:31:02.712 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1200 IOs, 438261760 bytes 2026-02-22T20:31:03.280 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1300 IOs, 460241920 bytes 2026-02-22T20:31:03.634 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:1400 IOs, 490290176 bytes 2026-02-22T20:31:03.752 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:31:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:31:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:31:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:31:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2026-02-22T20:31:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:31:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:31:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:31:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:31:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:31:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:31:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_8 2026-02-22T20:31:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:31:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:31:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:31:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-22T20:31:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:31:03.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a5d865ab7 2026-02-22T20:31:03.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:31:03.785 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:31:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_8 present 12a5d865ab7 2026-02-22T20:31:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:31:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:31:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:31:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a5d865ab7 2026-02-22T20:31:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:31:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:31:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_8 2026-02-22T20:31:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:31:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:31:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:31:03.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-22T20:31:03.888 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a5d865ab7 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a5d865ab7 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a5d865ab7 -o 12a5d865ab7 = 12a5d865ab7 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_8 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:31:03.920 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2026-02-22T20:31:05.585 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2026-02-22T20:31:05.586 INFO:teuthology.orchestra.run:waiting for 600 2026-02-22T20:31:05.586 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2026-02-22T20:31:05.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-22T20:31:05.587 DEBUG:teuthology.orchestra.run.trial066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-22T20:31:05.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-22T20:31:05.590 INFO:tasks.rbd_mirror_thrash:waiting for 117.67480652791424 secs before thrashing 2026-02-22T20:31:34.031 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_8 2026-02-22T20:31:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:31:34.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:31:34.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:31:34.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-22T20:31:34.041 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:31:34.066 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_8: (2) No such file or directory 2026-02-22T20:31:34.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:31:34.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:31:34.069 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:31:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2026-02-22T20:31:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:31:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:31:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:31:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:31:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:31:34.170 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:31:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_8 2026-02-22T20:31:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:31:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:31:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_8 2026-02-22T20:31:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-22T20:31:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:31:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_8: (2) No such file or directory 2026-02-22T20:31:34.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:31:34.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:31:34.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:31:34.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:31:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_8 2026-02-22T20:31:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:31:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:31:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2026-02-22T20:31:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_8 2026-02-22T20:31:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:31:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:31:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:31:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:31:34.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_8 2026-02-22T20:31:34.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:31:34.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:34.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_8 2026-02-22T20:31:34.233 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2026-02-22T20:31:34.263 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2026-02-22T20:31:34.407 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:31:34.412 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:31:34.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:31:34.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_9 2026-02-22T20:31:34.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_9 512M 2026-02-22T20:31:34.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:31:34.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:31:34.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:34.413 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:31:34.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:31:34.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:31:34.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:31:34.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:31:34.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:31:34.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:31:34.414 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_9 2026-02-22T20:31:34.451 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_9 2026-02-22T20:31:34.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:31:34.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:34.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:31:34.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:31:34.452 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 journal 2026-02-22T20:31:34.481 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:31:34.487 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2026-02-22T20:31:34.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_9': 2026-02-22T20:31:34.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:31:34.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:31:34.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:31:34.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 38aee0ecfca1 2026-02-22T20:31:34.514 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.38aee0ecfca1 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:31:34 2026 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:31:34 2026 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:31:34 2026 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 38aee0ecfca1 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: f07d7e59-4395-4abf-a450-84e481b7f331 2026-02-22T20:31:34.516 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:31:34.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_9 2026-02-22T20:31:34.517 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:31:34.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:34.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:31:34.518 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:31:34.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=6 2026-02-22T20:31:34.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:31:34.519 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:31:34.647 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 594944 bytes 2026-02-22T20:31:36.246 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 103013376 bytes 2026-02-22T20:31:38.836 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 167292928 bytes 2026-02-22T20:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 209145856 bytes 2026-02-22T20:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_9 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-22T20:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:31:40.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a51793baf9 2026-02-22T20:31:40.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:31:40.763 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:31:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_9 present 12a51793baf9 2026-02-22T20:31:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:31:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:31:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:31:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a51793baf9 2026-02-22T20:31:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:31:40.865 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:31:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_9 2026-02-22T20:31:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:31:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:31:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:31:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-22T20:31:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:31:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a51793baf9 2026-02-22T20:31:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a51793baf9 2026-02-22T20:31:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a51793baf9 -o 12a51793baf9 = 12a51793baf9 2026-02-22T20:31:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:31:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:31:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:31:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_9 2026-02-22T20:31:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:31:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:31:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:31:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2026-02-22T20:32:08.484 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9553997525495779) > weight (0.25) 2026-02-22T20:32:08.485 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5724121785680873) > weight (0.25) 2026-02-22T20:32:08.485 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.48934830863734746) > weight (0.25) 2026-02-22T20:32:08.485 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5035712016104353) > weight (0.25) 2026-02-22T20:32:08.485 INFO:tasks.rbd_mirror_thrash:waiting for 98.89881769130992 secs before thrashing 2026-02-22T20:32:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:32:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2026-02-22T20:32:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:32:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:32:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:32:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:32:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:32:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_9 2026-02-22T20:32:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:32:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-22T20:32:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:32:10.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_9: (2) No such file or directory 2026-02-22T20:32:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:32:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:32:10.223 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_9 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_9 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-22T20:32:10.325 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:32:10.346 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_9: (2) No such file or directory 2026-02-22T20:32:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:32:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:32:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:32:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_9 2026-02-22T20:32:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:32:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2026-02-22T20:32:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_9 2026-02-22T20:32:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:32:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:32:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:32:10.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_9 2026-02-22T20:32:10.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:10.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:10.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_9 2026-02-22T20:32:10.383 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2026-02-22T20:32:10.415 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2026-02-22T20:32:10.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for i in `seq 1 10` 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image=image_10 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ create_image cluster2 mirror image_10 512M 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local size=128 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -n 512M ']' 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ size=512M 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ shift 2026-02-22T20:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_10 2026-02-22T20:32:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ enable_mirror cluster2 mirror image_10 2026-02-22T20:32:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:10.601 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local mode=journal 2026-02-22T20:32:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 journal 2026-02-22T20:32:10.632 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring enabled 2026-02-22T20:32:10.638 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:rbd image 'image_10': 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: size 512 MiB in 128 objects 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: order 22 (4 MiB objects) 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: snapshot_count: 0 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: id: 38bee0b02cb5 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: block_name_prefix: rbd_data.38bee0b02cb5 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: format: 2 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: op_features: 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: flags: 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: create_timestamp: Sun Feb 22 20:32:10 2026 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: access_timestamp: Sun Feb 22 20:32:10 2026 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: modify_timestamp: Sun Feb 22 20:32:10 2026 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: journal: 38bee0b02cb5 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring state: enabled 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring mode: journal 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring global id: 9c2ada17-9918-484a-b349-dea179f69fcf 2026-02-22T20:32:10.669 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout: mirroring primary: true 2026-02-22T20:32:10.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stress_write_image cluster2 mirror image_10 2026-02-22T20:32:10.672 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-22T20:32:10.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local duration=6 2026-02-22T20:32:10.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:32:10.675 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2026-02-22T20:32:10.805 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:100 IOs, 24235008 bytes 2026-02-22T20:32:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:200 IOs, 53455872 bytes 2026-02-22T20:32:12.270 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:300 IOs, 94734336 bytes 2026-02-22T20:32:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:400 IOs, 120168448 bytes 2026-02-22T20:32:14.686 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:500 IOs, 156125184 bytes 2026-02-22T20:32:16.090 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:600 IOs, 228458496 bytes 2026-02-22T20:32:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=124 2026-02-22T20:32:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:32:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 124 -eq 124 ']' 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=present 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_10 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-22T20:32:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:32:16.722 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id=12a563c0a785 2026-02-22T20:32:16.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:32:16.723 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:32:16.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_10 present 12a563c0a785 2026-02-22T20:32:16.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:16.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:16.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:16.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=present 2026-02-22T20:32:16.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id=12a563c0a785 2026-02-22T20:32:16.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:32:16.824 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:32:16.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_10 2026-02-22T20:32:16.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:16.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:16.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:32:16.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-22T20:32:16.825 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:32:16.855 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id=12a563c0a785 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 12a563c0a785 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -z 12a563c0a785 -o 12a563c0a785 = 12a563c0a785 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_state=present 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test present = present 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ disable_mirror cluster2 mirror image_10 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:16.856 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2026-02-22T20:32:47.017 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:Mirroring disabled 2026-02-22T20:32:47.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2026-02-22T20:32:47.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:47.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:47.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local state=deleted 2026-02-22T20:32:47.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:32:47.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local s 2026-02-22T20:32:47.026 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:32:47.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_10 2026-02-22T20:32:47.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:47.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:32:47.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-22T20:32:47.027 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:32:47.053 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_10: (2) No such file or directory 2026-02-22T20:32:47.056 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ image_id= 2026-02-22T20:32:47.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:32:47.057 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0.1 2026-02-22T20:32:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2026-02-22T20:32:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local test_state=deleted 2026-02-22T20:32:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image_id= 2026-02-22T20:32:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_state=deleted 2026-02-22T20:32:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local current_image_id 2026-02-22T20:32:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ get_image_id cluster1 mirror image_10 2026-02-22T20:32:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local image=image_10 2026-02-22T20:32:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-22T20:32:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:32:47.182 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:rbd: error opening image image_10: (2) No such file or directory 2026-02-22T20:32:47.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ current_image_id= 2026-02-22T20:32:47.185 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n '' 2026-02-22T20:32:47.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test deleted = deleted 2026-02-22T20:32:47.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:47.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ purge_snapshots cluster2 mirror image_10 2026-02-22T20:32:47.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:47.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:47.186 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2026-02-22T20:32:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image_retry cluster2 mirror image_10 2026-02-22T20:32:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:32:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:32:47.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ remove_image cluster2 mirror image_10 2026-02-22T20:32:47.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:47.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local image=image_10 2026-02-22T20:32:47.221 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2026-02-22T20:32:47.252 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2026-02-22T20:32:47.393 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:32:47.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:47.397 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2026-02-22T20:32:47.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ tee -a /tmp/tmp.Xb7Kt1LMAh/rbd-mirror.test.log 2026-02-22T20:32:47.398 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ date '+%F %T' 2026-02-22T20:32:47.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo 2026-02-22 20:32:47 TEST: check if removed 'images'\''' OMAP are removed 2026-02-22T20:32:47.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:2026-02-22 20:32:47 TEST: check if removed images' OMAP are removed 2026-02-22T20:32:47.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_in_omap cluster1 mirror 2026-02-22T20:32:47.400 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2026-02-22T20:32:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local obj_name=rbd_mirroring 2026-02-22T20:32:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local filter=status_global 2026-02-22T20:32:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:32:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:32:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local filter=status_global 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -c status_global 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:32:47.402 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 = 0 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local obj_name=rbd_mirroring 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local filter=image_ 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:32:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:32:47.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:32:47.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2026-02-22T20:32:47.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:47.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:32:47.435 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local filter=image_ 2026-02-22T20:32:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-22T20:32:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:32:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -c image_ 2026-02-22T20:32:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-22T20:32:47.461 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 = 0 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local obj_name=rbd_mirror_leader 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local filter=image_map 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:32:47.462 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:32:47.463 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local filter=image_map 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster1 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-22T20:32:47.464 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -c image_map 2026-02-22T20:32:47.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 = 0 2026-02-22T20:32:47.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T20:32:47.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:32:47.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:32:47.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:47.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_image_in_omap cluster2 mirror 2026-02-22T20:32:47.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:47.488 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local obj_name=rbd_mirroring 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local filter=status_global 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local filter=status_global 2026-02-22T20:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-22T20:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -c status_global 2026-02-22T20:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 = 0 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local obj_name=rbd_mirroring 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local filter=image_ 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:32:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:32:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:32:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2026-02-22T20:32:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:32:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:32:47.512 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local filter=image_ 2026-02-22T20:32:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-22T20:32:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:32:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -c image_ 2026-02-22T20:32:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:32:47.513 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 = 0 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local pool=mirror 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local obj_name=rbd_mirror_leader 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local filter=image_map 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:32:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ sleep 0 2026-02-22T20:32:47.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:32:47.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2026-02-22T20:32:47.535 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:32:47.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:32:47.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local filter=image_map 2026-02-22T20:32:47.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2026-02-22T20:32:47.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local cluster=cluster2 2026-02-22T20:32:47.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local pool=mirror 2026-02-22T20:32:47.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:32:47.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2026-02-22T20:32:47.536 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ grep -c image_map 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test 0 = 0 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ error_code=0 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set -e 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 0 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ cleanup 0 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local error_code=0 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ set +e 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -ne 0 ']' 2026-02-22T20:32:47.557 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -z '' ']' 2026-02-22T20:32:47.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster instance 2026-02-22T20:32:47.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T20:32:47.558 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2026-02-22T20:32:48.308 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:pool 'mirror' does not exist 2026-02-22T20:32:48.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T20:32:48.317 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2026-02-22T20:32:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:pool 'mirror' does not exist 2026-02-22T20:32:48.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T20:32:48.978 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2026-02-22T20:32:49.312 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:pool 'mirror_parent' does not exist 2026-02-22T20:32:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ CEPH_ARGS= 2026-02-22T20:32:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2026-02-22T20:32:49.976 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:pool 'mirror_parent' does not exist 2026-02-22T20:32:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2026-02-22T20:32:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirrors cluster1 2026-02-22T20:32:49.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1 2026-02-22T20:32:49.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:49.994 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:32:49.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10a969b640 -1 rbd::mirror::RemotePoolPoller: 0x5646e2afaa90 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2026-02-22T20:32:49.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10a8e9a640 -1 rbd::mirror::RemotePoolPoller: 0x5646e2afaa90 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2026-02-22T20:32:49.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10a8e9a640 -1 rbd::mirror::PoolReplayer: 0x5646e3a08000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2026-02-22T20:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirror cluster1:0 '' 2026-02-22T20:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:0 2026-02-22T20:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T20:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T20:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirror cluster1:1 '' 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:1 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirror cluster1:2 '' 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:2 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T20:32:49.999 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirror cluster1:3 '' 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster1:3 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirrors cluster2 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:++ seq 0 3 2026-02-22T20:32:50.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10b66b5640 -1 rbd::mirror::InstanceWatcher: 0x5646e3c02700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14384 object: (2) No such file or directory 2026-02-22T20:32:50.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10b66b5640 -1 rbd::mirror::InstanceWatcher: 0x5646e3c02a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14384 object: (2) No such file or directory 2026-02-22T20:32:50.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10afea8640 -1 rbd::mirror::InstanceWatcher: 0x5646e3c02700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10afea8640 -1 rbd::mirror::InstanceWatcher: 0x5646e3c02a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10a969b640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e3cf8360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10b66b5640 -1 rbd::mirror::NamespaceReplayer: 0x5646e3cf6000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10af6a7640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e3cf8000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10b5eb4640 -1 rbd::mirror::NamespaceReplayer: 0x5646e3cf6000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10a8e9a640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e3cf8480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10b66b5640 -1 rbd::mirror::NamespaceReplayer: 0x5646e3cf61a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10afea8640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e3cf8120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:49.988+0000 7f10b5eb4640 -1 rbd::mirror::NamespaceReplayer: 0x5646e3cf61a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:32:50.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirror cluster2:0 '' 2026-02-22T20:32:50.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2:0 2026-02-22T20:32:50.005 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirror cluster2:1 '' 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2:1 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirror cluster2:2 '' 2026-02-22T20:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2:2 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ stop_mirror cluster2:3 '' 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local cluster=cluster2:3 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ local sig= 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test -n 1 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ return 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' -z 1 ']' 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ test '' = /tmp/tmp.Xb7Kt1LMAh 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ rm -Rf /tmp/tmp.Xb7Kt1LMAh 2026-02-22T20:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:32:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ echo OK 2026-02-22T20:32:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial066.stdout:OK 2026-02-22T20:32:50.009 INFO:tasks.workunit.cluster1.client.mirror.trial066.stderr:+ exit 0 2026-02-22T20:32:50.010 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T20:32:50.010 DEBUG:teuthology.orchestra.run.trial066:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2026-02-22T20:32:50.062 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2026-02-22T20:32:50.063 DEBUG:teuthology.orchestra.run.trial066:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2026-02-22T20:32:50.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.068+0000 7f109be80640 -1 rbd::mirror::RemotePoolPoller: 0x5646e2afa820 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2026-02-22T20:32:50.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.068+0000 7f109b67f640 -1 rbd::mirror::RemotePoolPoller: 0x5646e2afa820 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2026-02-22T20:32:50.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.068+0000 7f109b67f640 -1 rbd::mirror::PoolReplayer: 0x5646e3cae000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2026-02-22T20:32:50.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.068+0000 7f10b5eb4640 -1 rbd::mirror::InstanceWatcher: 0x5646e40b0e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14387 object: (2) No such file or directory 2026-02-22T20:32:50.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.068+0000 7f10a268d640 -1 rbd::mirror::InstanceWatcher: 0x5646e40b0e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.068+0000 7f109be80640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e3f20b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.068+0000 7f10b5eb4640 -1 rbd::mirror::NamespaceReplayer: 0x5646e4172000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.072+0000 7f10a1e8c640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e3f20fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.072+0000 7f10b66b5640 -1 rbd::mirror::NamespaceReplayer: 0x5646e4172000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.100+0000 7f10b5eb4640 -1 rbd::mirror::LeaderWatcher: 0x5646e46efb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.100+0000 7f10b5eb4640 -1 rbd::mirror::InstanceWatcher: 0x5646e3c02380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14384 object: (2) No such file or directory 2026-02-22T20:32:50.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.100+0000 7f10af6a7640 -1 rbd::mirror::InstanceWatcher: 0x5646e3c02380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.100+0000 7f10a969b640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e46dc6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.100+0000 7f10b66b5640 -1 rbd::mirror::NamespaceReplayer: 0x5646e39e81a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.100+0000 7f10afea8640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e46db440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.100+0000 7f10b5eb4640 -1 rbd::mirror::NamespaceReplayer: 0x5646e39e81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.104+0000 7f10b5eb4640 -1 rbd::mirror::LeaderWatcher: 0x5646e40a8300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.104+0000 7f10b5eb4640 -1 rbd::mirror::InstanceWatcher: 0x5646e40b0380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14387 object: (2) No such file or directory 2026-02-22T20:32:50.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.104+0000 7f10a268d640 -1 rbd::mirror::InstanceWatcher: 0x5646e40b0380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.104+0000 7f109b67f640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e3f206c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.104+0000 7f10b5eb4640 -1 rbd::mirror::NamespaceReplayer: 0x5646e3cf64e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.104+0000 7f10a1e8c640 -1 rbd::mirror::MirrorStatusUpdater 0x5646e3f137a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.104+0000 7f10b66b5640 -1 rbd::mirror::NamespaceReplayer: 0x5646e3cf64e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.328 DEBUG:teuthology.parallel:result is None 2026-02-22T20:32:50.328 DEBUG:teuthology.orchestra.run.trial066:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-22T20:32:50.378 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-22T20:32:50.379 DEBUG:teuthology.orchestra.run.trial066:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-22T20:32:50.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.384+0000 7f820f431640 -1 rbd::mirror::PoolWatcher: 0x55a3b062e1e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.384+0000 7f820f431640 -1 rbd::mirror::PoolWatcher: 0x55a3b062fa40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.384+0000 7f8229465640 -1 rbd::mirror::LeaderWatcher: 0x55a3b059a300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2026-02-22T20:32:50.395 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-22T20:32:50.395 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2026-02-22T20:32:50.407 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2026-02-22T20:32:50.407 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 17 2026-02-22T20:32:50.407 INFO:tasks.rbd_mirror_thrash:done joining 2026-02-22T20:32:50.408 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2026-02-22T20:32:50.428 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2026-02-22T20:32:50.428 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 15 2026-02-22T20:32:50.428 INFO:tasks.rbd_mirror_thrash:done joining 2026-02-22T20:32:50.428 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-22T20:32:50.446 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2026-02-22T20:32:50.446 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:32:50.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.440+0000 7f10b6eb6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 129076) UID: 1000 2026-02-22T20:32:50.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2026-02-22T20:32:50.458 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-22T20:32:50.476 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2026-02-22T20:32:50.476 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:32:50.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f68220640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 118009) UID: 1000 2026-02-22T20:32:50.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f61212640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56251a62c6c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-22T20:32:50.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f61212640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56251a62cb40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-22T20:32:50.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f61212640 -1 rbd::mirror::PoolWatcher: 0x56251ad75680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f61212640 -1 rbd::mirror::PoolWatcher: 0x56251ae4a5a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f6721e640 -1 rbd::mirror::InstanceWatcher: 0x56251aa44700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13530 object: (2) No such file or directory 2026-02-22T20:32:50.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f5aa05640 -1 rbd::mirror::PoolWatcher: 0x56251ad74b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f5aa05640 -1 rbd::mirror::PoolWatcher: 0x56251ad752c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f531f6640 -1 rbd::mirror::InstanceWatcher: 0x56251aa44700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f6721e640 -1 rbd::mirror::InstanceWatcher: 0x56251aee3880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13528 object: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f6721e640 -1 rbd::mirror::InstanceWatcher: 0x56251aee3c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13528 object: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f61212640 -1 rbd::mirror::InstanceWatcher: 0x56251aee3880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f60a11640 -1 rbd::mirror::InstanceWatcher: 0x56251aee3c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f4c9e9640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a8a0a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f6721e640 -1 rbd::mirror::NamespaceReplayer: 0x56251ab00000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f539f7640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a8a0900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f67a1f640 -1 rbd::mirror::NamespaceReplayer: 0x56251ab00000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f5a204640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a6865a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f67a1f640 -1 rbd::mirror::NamespaceReplayer: 0x56251a684000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f5aa05640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a686360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f6721e640 -1 rbd::mirror::NamespaceReplayer: 0x56251a6841a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f61212640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a686000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f67a1f640 -1 rbd::mirror::NamespaceReplayer: 0x56251a684000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f60a11640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a686120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f67a1f640 -1 rbd::mirror::NamespaceReplayer: 0x56251a6841a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f67a1f640 -1 rbd::mirror::LeaderWatcher: 0x56251aa20300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f67a1f640 -1 rbd::mirror::InstanceWatcher: 0x56251aa44380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13530 object: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f531f6640 -1 rbd::mirror::InstanceWatcher: 0x56251aa44380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f4d1ea640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a686b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f67a1f640 -1 rbd::mirror::NamespaceReplayer: 0x56251a6844e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f539f7640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a891b00 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.496+0000 7f9f6721e640 -1 rbd::mirror::NamespaceReplayer: 0x56251a6844e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f61212640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56251ad1c900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-22T20:32:50.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f60a11640 -1 rbd::mirror::PoolWatcher: 0x56251ad75e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f5a204640 -1 rbd::mirror::PoolWatcher: 0x56251ad75860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f6721e640 -1 rbd::mirror::LeaderWatcher: 0x56251b079b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2026-02-22T20:32:50.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f6721e640 -1 rbd::mirror::LeaderWatcher: 0x56251b079b00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2026-02-22T20:32:50.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f67a1f640 -1 rbd::mirror::LeaderWatcher: 0x56251b079b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f67a1f640 -1 rbd::mirror::InstanceWatcher: 0x56251aee3500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13528 object: (2) No such file or directory 2026-02-22T20:32:50.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f61212640 -1 rbd::mirror::InstanceWatcher: 0x56251aee3500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f5aa05640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a578480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f6721e640 -1 rbd::mirror::NamespaceReplayer: 0x56251a3761a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f60a11640 -1 rbd::mirror::MirrorStatusUpdater 0x56251a3dd560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.500+0000 7f9f67a1f640 -1 rbd::mirror::NamespaceReplayer: 0x56251a3761a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2026-02-22T20:32:50.524 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-22T20:32:50.542 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2026-02-22T20:32:50.542 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:32:50.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c97edc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 130418) UID: 1000 2026-02-22T20:32:50.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c976db640 -1 rbd::mirror::InstanceWatcher: 0x55e918d33500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14489 object: (2) No such file or directory 2026-02-22T20:32:50.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c976db640 -1 rbd::mirror::InstanceWatcher: 0x55e918d33880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14489 object: (2) No such file or directory 2026-02-22T20:32:50.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c96eda640 -1 rbd::mirror::InstanceWatcher: 0x55e918e76700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14494 object: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c836b3640 -1 rbd::mirror::InstanceWatcher: 0x55e918e76700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c906cd640 -1 rbd::mirror::InstanceWatcher: 0x55e918d33500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c906cd640 -1 rbd::mirror::InstanceWatcher: 0x55e918d33880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c7c6a5640 -1 rbd::mirror::MirrorStatusUpdater 0x55e918ef07e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c976db640 -1 rbd::mirror::NamespaceReplayer: 0x55e91916e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c89ec0640 -1 rbd::mirror::MirrorStatusUpdater 0x55e918e267e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c89ec0640 -1 rbd::mirror::MirrorStatusUpdater 0x55e918e26360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c96eda640 -1 rbd::mirror::NamespaceReplayer: 0x55e918d7a1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c96eda640 -1 rbd::mirror::NamespaceReplayer: 0x55e918d7a340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c82eb2640 -1 rbd::mirror::MirrorStatusUpdater 0x55e918ef0fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c976db640 -1 rbd::mirror::NamespaceReplayer: 0x55e91916e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c90ece640 -1 rbd::mirror::MirrorStatusUpdater 0x55e918e26000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c90ece640 -1 rbd::mirror::MirrorStatusUpdater 0x55e918e26120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c96eda640 -1 rbd::mirror::NamespaceReplayer: 0x55e918d7a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c96eda640 -1 rbd::mirror::NamespaceReplayer: 0x55e918d7a340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c96eda640 -1 rbd::mirror::LeaderWatcher: 0x55e918f04c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c96eda640 -1 rbd::mirror::InstanceWatcher: 0x55e918d33c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14494 object: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c836b3640 -1 rbd::mirror::InstanceWatcher: 0x55e918d33c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c7cea6640 -1 rbd::mirror::MirrorStatusUpdater 0x55e918ef06c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c96eda640 -1 rbd::mirror::NamespaceReplayer: 0x55e918d7a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c82eb2640 -1 rbd::mirror::MirrorStatusUpdater 0x55e918edb7a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.564+0000 7f4c976db640 -1 rbd::mirror::NamespaceReplayer: 0x55e918d7a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.568+0000 7f4c96eda640 -1 rbd::mirror::LeaderWatcher: 0x55e918a27b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.568+0000 7f4c96eda640 -1 rbd::mirror::InstanceWatcher: 0x55e919521500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14489 object: (2) No such file or directory 2026-02-22T20:32:50.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.568+0000 7f4c906cd640 -1 rbd::mirror::InstanceWatcher: 0x55e919521500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.568+0000 7f4c8a6c1640 -1 rbd::mirror::MirrorStatusUpdater 0x55e918a146c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.568+0000 7f4c976db640 -1 rbd::mirror::NamespaceReplayer: 0x55e9189ac1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.568+0000 7f4c90ece640 -1 rbd::mirror::MirrorStatusUpdater 0x55e919655320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.568+0000 7f4c96eda640 -1 rbd::mirror::NamespaceReplayer: 0x55e9189ac1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2026-02-22T20:32:50.589 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-22T20:32:50.609 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2026-02-22T20:32:50.609 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:32:50.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25072e7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 128187) UID: 1000 2026-02-22T20:32:50.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f22bd640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56081f064000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-22T20:32:50.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25062e5640 -1 rbd::mirror::InstanceWatcher: 0x56081f70f880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14323 object: (2) No such file or directory 2026-02-22T20:32:50.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25062e5640 -1 rbd::mirror::InstanceWatcher: 0x56081ef4b500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14323 object: (2) No such file or directory 2026-02-22T20:32:50.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f2abe640 -1 rbd::mirror::PoolWatcher: 0x56081f406b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24ebab0640 -1 rbd::mirror::PoolWatcher: 0x56081f39e960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25002d9640 -1 rbd::mirror::InstanceWatcher: 0x56081f70f880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24ffad8640 -1 rbd::mirror::InstanceWatcher: 0x56081ef4b500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25062e5640 -1 rbd::mirror::InstanceWatcher: 0x56081f2be380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14326 object: (2) No such file or directory 2026-02-22T20:32:50.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f9acc640 -1 rbd::mirror::MirrorStatusUpdater 0x56081eece360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f9acc640 -1 rbd::mirror::MirrorStatusUpdater 0x56081eece5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f2506ae6640 -1 rbd::mirror::NamespaceReplayer: 0x56081eebc000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f2506ae6640 -1 rbd::mirror::NamespaceReplayer: 0x56081eebc1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25002d9640 -1 rbd::mirror::MirrorStatusUpdater 0x56081eece000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25002d9640 -1 rbd::mirror::MirrorStatusUpdater 0x56081eece120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25062e5640 -1 rbd::mirror::NamespaceReplayer: 0x56081eebc000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25062e5640 -1 rbd::mirror::NamespaceReplayer: 0x56081eebc1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f2abe640 -1 rbd::mirror::InstanceWatcher: 0x56081f2be380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24ec2b1640 -1 rbd::mirror::MirrorStatusUpdater 0x56081f10ab40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f2506ae6640 -1 rbd::mirror::NamespaceReplayer: 0x56081f366000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f22bd640 -1 rbd::mirror::MirrorStatusUpdater 0x56081f10aa20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25062e5640 -1 rbd::mirror::NamespaceReplayer: 0x56081f366000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f2abe640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56081ee6a000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-22T20:32:50.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f22bd640 -1 rbd::mirror::PoolWatcher: 0x56081f39eb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24ebab0640 -1 rbd::mirror::PoolWatcher: 0x56081dc5e000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f2506ae6640 -1 rbd::mirror::LeaderWatcher: 0x56081f2a8300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2026-02-22T20:32:50.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f2506ae6640 -1 rbd::mirror::LeaderWatcher: 0x56081f2a8300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2026-02-22T20:32:50.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25062e5640 -1 rbd::mirror::LeaderWatcher: 0x56081f2a8300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25062e5640 -1 rbd::mirror::InstanceWatcher: 0x56081ef4b880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14326 object: (2) No such file or directory 2026-02-22T20:32:50.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f2abe640 -1 rbd::mirror::InstanceWatcher: 0x56081ef4b880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24ec2b1640 -1 rbd::mirror::MirrorStatusUpdater 0x56081eece6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f2506ae6640 -1 rbd::mirror::NamespaceReplayer: 0x56081eebc4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f24f22bd640 -1 rbd::mirror::MirrorStatusUpdater 0x56081f0a39e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.624+0000 7f25062e5640 -1 rbd::mirror::NamespaceReplayer: 0x56081eebc4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.628+0000 7f2506ae6640 -1 rbd::mirror::LeaderWatcher: 0x56081ec25b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.628+0000 7f2506ae6640 -1 rbd::mirror::InstanceWatcher: 0x56081f70f500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.14323 object: (2) No such file or directory 2026-02-22T20:32:50.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.628+0000 7f24ffad8640 -1 rbd::mirror::InstanceWatcher: 0x56081f70f500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.628+0000 7f24f92cb640 -1 rbd::mirror::MirrorStatusUpdater 0x56081edb8480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.628+0000 7f25062e5640 -1 rbd::mirror::NamespaceReplayer: 0x56081ebac340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.628+0000 7f25002d9640 -1 rbd::mirror::MirrorStatusUpdater 0x56081ec23560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.628+0000 7f2506ae6640 -1 rbd::mirror::NamespaceReplayer: 0x56081ebac340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2026-02-22T20:32:50.650 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-22T20:32:50.669 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2026-02-22T20:32:50.670 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:32:50.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f822a467640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 127589) UID: 1000 2026-02-22T20:32:50.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f821543d640 -1 rbd::mirror::InstanceWatcher: 0x55a3b0648380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8229465640 -1 rbd::mirror::InstanceWatcher: 0x55a3b0648700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5277 object: (2) No such file or directory 2026-02-22T20:32:50.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8229465640 -1 rbd::mirror::InstanceWatcher: 0x55a3b031f880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5277 object: (2) No such file or directory 2026-02-22T20:32:50.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f820ec30640 -1 rbd::mirror::MirrorStatusUpdater 0x55a3b047f0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8229c66640 -1 rbd::mirror::NamespaceReplayer: 0x55a3b06f8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8215c3e640 -1 rbd::mirror::MirrorStatusUpdater 0x55a3b047efc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8229465640 -1 rbd::mirror::NamespaceReplayer: 0x55a3b06f8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8223459640 -1 rbd::mirror::InstanceWatcher: 0x55a3b0648700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8222c58640 -1 rbd::mirror::InstanceWatcher: 0x55a3b031f880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8223459640 -1 rbd::mirror::MirrorStatusUpdater 0x55a3b0298000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8229465640 -1 rbd::mirror::NamespaceReplayer: 0x55a3b0296000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8222c58640 -1 rbd::mirror::MirrorStatusUpdater 0x55a3b0298120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8229465640 -1 rbd::mirror::NamespaceReplayer: 0x55a3b02961a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f821543d640 -1 rbd::mirror::InstanceWatcher: 0x55a3b031fc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f820f431640 -1 rbd::mirror::MirrorStatusUpdater 0x55a3b047ea20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.688+0000 7f8229465640 -1 rbd::mirror::NamespaceReplayer: 0x55a3b02964e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.692+0000 7f8229c66640 -1 rbd::mirror::LeaderWatcher: 0x55a3affddb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.692+0000 7f8229c66640 -1 rbd::mirror::InstanceWatcher: 0x55a3b0adb500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5277 object: (2) No such file or directory 2026-02-22T20:32:50.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.692+0000 7f8223459640 -1 rbd::mirror::InstanceWatcher: 0x55a3b0adb500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.692+0000 7f821c44b640 -1 rbd::mirror::MirrorStatusUpdater 0x55a3b0c786c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.692+0000 7f8229c66640 -1 rbd::mirror::NamespaceReplayer: 0x55a3aff6e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.692+0000 7f8222c58640 -1 rbd::mirror::MirrorStatusUpdater 0x55a3b0c05320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial066.stderr:2026-02-22T20:32:50.692+0000 7f8229465640 -1 rbd::mirror::NamespaceReplayer: 0x55a3aff6e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2026-02-22T20:32:50.713 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-22T20:32:50.732 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2026-02-22T20:32:50.732 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:32:50.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef32b3b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 101046) UID: 1000 2026-02-22T20:32:50.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef31b39640 -1 rbd::mirror::InstanceWatcher: 0x55d2c4d4f880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4773 object: (2) No such file or directory 2026-02-22T20:32:50.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef31b39640 -1 rbd::mirror::InstanceWatcher: 0x55d2c48b0700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4776 object: (2) No such file or directory 2026-02-22T20:32:50.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef31b39640 -1 rbd::mirror::InstanceWatcher: 0x55d2c4d4fc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4773 object: (2) No such file or directory 2026-02-22T20:32:50.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef2b32c640 -1 rbd::mirror::InstanceWatcher: 0x55d2c4d4f880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef1db11640 -1 rbd::mirror::InstanceWatcher: 0x55d2c48b0700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef2bb2d640 -1 rbd::mirror::InstanceWatcher: 0x55d2c4d4fc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef24b1f640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c44fa360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef24b1f640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c44fa5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef3233a640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c44f81a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef3233a640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c44f8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef17304640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c44fac60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef31b39640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c49544e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef2b32c640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c44fa120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef3233a640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c44f81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef2bb2d640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c44fa000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef3233a640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c44f8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef1e312640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c46e0a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef3233a640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c49544e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef31b39640 -1 rbd::mirror::LeaderWatcher: 0x55d2c4730c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef31b39640 -1 rbd::mirror::InstanceWatcher: 0x55d2c48b0380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4776 object: (2) No such file or directory 2026-02-22T20:32:50.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef1db11640 -1 rbd::mirror::InstanceWatcher: 0x55d2c48b0380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef17304640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c44fab40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef3233a640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c4954000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef1e312640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c46c18c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.748+0000 7fef31b39640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c4954000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.752+0000 7fef31b39640 -1 rbd::mirror::LeaderWatcher: 0x55d2c4ef7b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.752+0000 7fef31b39640 -1 rbd::mirror::InstanceWatcher: 0x55d2c4d4f500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4773 object: (2) No such file or directory 2026-02-22T20:32:50.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.752+0000 7fef2b32c640 -1 rbd::mirror::InstanceWatcher: 0x55d2c4d4f500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.752+0000 7fef25320640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c4ee86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.752+0000 7fef31b39640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c41e61a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.752+0000 7fef2bb2d640 -1 rbd::mirror::MirrorStatusUpdater 0x55d2c4e75200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial066.stderr:2026-02-22T20:32:50.752+0000 7fef3233a640 -1 rbd::mirror::NamespaceReplayer: 0x55d2c41e61a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2026-02-22T20:32:50.777 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-22T20:32:50.796 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2026-02-22T20:32:50.796 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:32:50.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa936ca640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 129748) UID: 1000 2026-02-22T20:32:50.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa8bebb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5566b1748000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa8bebb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5566b1534000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa8c6bc640 -1 rbd::mirror::PoolWatcher: 0x5566b196f0e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa8bebb640 -1 rbd::mirror::PoolWatcher: 0x5566b0390000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa7e6a0640 -1 rbd::mirror::InstanceWatcher: 0x5566b1e07c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa92ec9640 -1 rbd::mirror::InstanceWatcher: 0x5566b1e07880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5380 object: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa78694640 -1 rbd::mirror::MirrorStatusUpdater 0x5566b17b8c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa926c8640 -1 rbd::mirror::NamespaceReplayer: 0x5566b1a36000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa92ec9640 -1 rbd::mirror::InstanceWatcher: 0x5566b15e3180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5380 object: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa926c8640 -1 rbd::mirror::LeaderWatcher: 0x5566b18d0300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa8c6bc640 -1 rbd::mirror::InstanceWatcher: 0x5566b1e07880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa8bebb640 -1 rbd::mirror::InstanceWatcher: 0x5566b15e3180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa8c6bc640 -1 rbd::mirror::MirrorStatusUpdater 0x5566b1584240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa92ec9640 -1 rbd::mirror::NamespaceReplayer: 0x5566b15b8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa8bebb640 -1 rbd::mirror::MirrorStatusUpdater 0x5566b1584360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa92ec9640 -1 rbd::mirror::NamespaceReplayer: 0x5566b15b81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa926c8640 -1 rbd::mirror::InstanceWatcher: 0x5566b1994380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5383 object: (2) No such file or directory 2026-02-22T20:32:50.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa7eea1640 -1 rbd::mirror::InstanceWatcher: 0x5566b1994380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa77e93640 -1 rbd::mirror::MirrorStatusUpdater 0x5566b17b87e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa926c8640 -1 rbd::mirror::NamespaceReplayer: 0x5566b15b84e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa7e6a0640 -1 rbd::mirror::MirrorStatusUpdater 0x5566b1771560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.816+0000 7faa92ec9640 -1 rbd::mirror::NamespaceReplayer: 0x5566b15b84e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.820+0000 7faa8c6bc640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5566b1981440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-22T20:32:50.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.820+0000 7faa8bebb640 -1 rbd::mirror::PoolWatcher: 0x5566b1aa2960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-22T20:32:50.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.820+0000 7faa92ec9640 -1 rbd::mirror::LeaderWatcher: 0x5566b1fb1b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2026-02-22T20:32:50.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.820+0000 7faa92ec9640 -1 rbd::mirror::LeaderWatcher: 0x5566b1fb1b00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2026-02-22T20:32:50.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.820+0000 7faa926c8640 -1 rbd::mirror::LeaderWatcher: 0x5566b1fb1b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.820+0000 7faa926c8640 -1 rbd::mirror::InstanceWatcher: 0x5566b1e07500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5380 object: (2) No such file or directory 2026-02-22T20:32:50.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.820+0000 7faa8c6bc640 -1 rbd::mirror::InstanceWatcher: 0x5566b1e07500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.820+0000 7faa8bebb640 -1 rbd::mirror::MirrorStatusUpdater 0x5566b1f29200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial066.stderr:2026-02-22T20:32:50.820+0000 7faa926c8640 -1 rbd::mirror::NamespaceReplayer: 0x5566b12aa1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2026-02-22T20:32:50.843 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-22T20:32:50.862 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2026-02-22T20:32:50.862 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:32:50.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c6250640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 130600) UID: 1000 2026-02-22T20:32:50.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c524e640 -1 rbd::mirror::InstanceWatcher: 0x562480cdd880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5424 object: (2) No such file or directory 2026-02-22T20:32:50.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c5a4f640 -1 rbd::mirror::InstanceWatcher: 0x562480cddc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5424 object: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c5a4f640 -1 rbd::mirror::InstanceWatcher: 0x56248082a700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5427 object: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10bea41640 -1 rbd::mirror::InstanceWatcher: 0x562480cdd880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10bf242640 -1 rbd::mirror::InstanceWatcher: 0x562480cddc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10b1226640 -1 rbd::mirror::InstanceWatcher: 0x56248082a700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10bea41640 -1 rbd::mirror::MirrorStatusUpdater 0x56248047c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c5a4f640 -1 rbd::mirror::NamespaceReplayer: 0x56248047a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10bf242640 -1 rbd::mirror::MirrorStatusUpdater 0x56248047c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c524e640 -1 rbd::mirror::NamespaceReplayer: 0x56248047a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10aaa19640 -1 rbd::mirror::MirrorStatusUpdater 0x56248047cc60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c524e640 -1 rbd::mirror::NamespaceReplayer: 0x5624808ce4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10b1a27640 -1 rbd::mirror::MirrorStatusUpdater 0x56248065ea20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c5a4f640 -1 rbd::mirror::NamespaceReplayer: 0x5624808ce4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c5a4f640 -1 rbd::mirror::LeaderWatcher: 0x562480774300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c5a4f640 -1 rbd::mirror::InstanceWatcher: 0x56248082a380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5427 object: (2) No such file or directory 2026-02-22T20:32:50.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10b1226640 -1 rbd::mirror::InstanceWatcher: 0x56248082a380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10ab21a640 -1 rbd::mirror::MirrorStatusUpdater 0x56248047cb40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c524e640 -1 rbd::mirror::NamespaceReplayer: 0x5624808ce000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10b1a27640 -1 rbd::mirror::MirrorStatusUpdater 0x56248062d8c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.880+0000 7f10c5a4f640 -1 rbd::mirror::NamespaceReplayer: 0x5624808ce000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.884+0000 7f10c5a4f640 -1 rbd::mirror::LeaderWatcher: 0x562480e85b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-22T20:32:50.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.884+0000 7f10c5a4f640 -1 rbd::mirror::InstanceWatcher: 0x562480cdd500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5424 object: (2) No such file or directory 2026-02-22T20:32:50.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.884+0000 7f10bea41640 -1 rbd::mirror::InstanceWatcher: 0x562480cdd500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-22T20:32:50.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.884+0000 7f10b8a35640 -1 rbd::mirror::MirrorStatusUpdater 0x5624801d85a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.884+0000 7f10c5a4f640 -1 rbd::mirror::NamespaceReplayer: 0x5624801801a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.884+0000 7f10bf242640 -1 rbd::mirror::MirrorStatusUpdater 0x562480e31320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-22T20:32:50.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial066.stderr:2026-02-22T20:32:50.884+0000 7f10c524e640 -1 rbd::mirror::NamespaceReplayer: 0x5624801801a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-22T20:32:50.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2026-02-22T20:32:50.906 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-22T20:32:50.928 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2026-02-22T20:32:50.929 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:32:51.093 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T20:32:51.093 INFO:teuthology.orchestra.run.trial066.stderr:dumped all 2026-02-22T20:32:51.103 INFO:teuthology.orchestra.run.trial066.stdout:{"pg_ready":true,"pg_map":{"version":1627,"stamp":"2026-02-22T20:32:49.926899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4392268,"kb_used_data":970844,"kb_used_omap":513,"kb_used_meta":3420862,"kb_avail":278723252,"statfs":{"total":289910292480,"available":285412610048,"internally_reserved":0,"allocated":994144256,"data_stored":984708791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":526019,"internal_metadata":3502963005},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-725831795,"num_objects":-188,"num_object_clones":0,"num_object_copies":-376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-188,"num_whiteouts":0,"num_read":-1990270,"num_read_kb":-78973599,"num_write":-1893668,"num_write_kb":-71753999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.103568"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.868606+0000","last_change":"2026-02-22T19:52:27.193486+0000","last_active":"2026-02-22T20:32:24.868606+0000","last_peered":"2026-02-22T20:32:24.868606+0000","last_clean":"2026-02-22T20:32:24.868606+0000","last_became_active":"2026-02-22T19:52:25.190488+0000","last_became_peered":"2026-02-22T19:52:25.190488+0000","last_unstale":"2026-02-22T20:32:24.868606+0000","last_undegraded":"2026-02-22T20:32:24.868606+0000","last_fullsized":"2026-02-22T20:32:24.868606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:38:23.678969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091298699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.868760+0000","last_change":"2026-02-22T19:52:27.192944+0000","last_active":"2026-02-22T20:32:24.868760+0000","last_peered":"2026-02-22T20:32:24.868760+0000","last_clean":"2026-02-22T20:32:24.868760+0000","last_became_active":"2026-02-22T19:52:25.190482+0000","last_became_peered":"2026-02-22T19:52:25.190482+0000","last_unstale":"2026-02-22T20:32:24.868760+0000","last_undegraded":"2026-02-22T20:32:24.868760+0000","last_fullsized":"2026-02-22T20:32:24.868760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:29:08.459932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037698099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.868667+0000","last_change":"2026-02-22T19:52:27.193158+0000","last_active":"2026-02-22T20:32:24.868667+0000","last_peered":"2026-02-22T20:32:24.868667+0000","last_clean":"2026-02-22T20:32:24.868667+0000","last_became_active":"2026-02-22T19:52:25.189560+0000","last_became_peered":"2026-02-22T19:52:25.189560+0000","last_unstale":"2026-02-22T20:32:24.868667+0000","last_undegraded":"2026-02-22T20:32:24.868667+0000","last_fullsized":"2026-02-22T20:32:24.868667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:07:26.737908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059989500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.866993+0000","last_change":"2026-02-22T19:52:27.192933+0000","last_active":"2026-02-22T20:32:24.866993+0000","last_peered":"2026-02-22T20:32:24.866993+0000","last_clean":"2026-02-22T20:32:24.866993+0000","last_became_active":"2026-02-22T19:52:25.189562+0000","last_became_peered":"2026-02-22T19:52:25.189562+0000","last_unstale":"2026-02-22T20:32:24.866993+0000","last_undegraded":"2026-02-22T20:32:24.866993+0000","last_fullsized":"2026-02-22T20:32:24.866993+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:33:55.622679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038538599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.882362+0000","last_change":"2026-02-22T19:52:27.190720+0000","last_active":"2026-02-22T20:32:24.882362+0000","last_peered":"2026-02-22T20:32:24.882362+0000","last_clean":"2026-02-22T20:32:24.882362+0000","last_became_active":"2026-02-22T19:52:25.189656+0000","last_became_peered":"2026-02-22T19:52:25.189656+0000","last_unstale":"2026-02-22T20:32:24.882362+0000","last_undegraded":"2026-02-22T20:32:24.882362+0000","last_fullsized":"2026-02-22T20:32:24.882362+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:44:53.529788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026044399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:25.109454+0000","last_change":"2026-02-22T19:52:27.516719+0000","last_active":"2026-02-22T20:32:25.109454+0000","last_peered":"2026-02-22T20:32:25.109454+0000","last_clean":"2026-02-22T20:32:25.109454+0000","last_became_active":"2026-02-22T19:52:25.189998+0000","last_became_peered":"2026-02-22T19:52:25.189998+0000","last_unstale":"2026-02-22T20:32:25.109454+0000","last_undegraded":"2026-02-22T20:32:25.109454+0000","last_fullsized":"2026-02-22T20:32:25.109454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:34:47.988806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026744600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:25.109725+0000","last_change":"2026-02-22T19:52:27.516715+0000","last_active":"2026-02-22T20:32:25.109725+0000","last_peered":"2026-02-22T20:32:25.109725+0000","last_clean":"2026-02-22T20:32:25.109725+0000","last_became_active":"2026-02-22T19:52:25.189993+0000","last_became_peered":"2026-02-22T19:52:25.189993+0000","last_unstale":"2026-02-22T20:32:25.109725+0000","last_undegraded":"2026-02-22T20:32:25.109725+0000","last_fullsized":"2026-02-22T20:32:25.109725+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:48:18.874126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027272700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.867534+0000","last_change":"2026-02-22T19:52:27.193332+0000","last_active":"2026-02-22T20:32:24.867534+0000","last_peered":"2026-02-22T20:32:24.867534+0000","last_clean":"2026-02-22T20:32:24.867534+0000","last_became_active":"2026-02-22T19:52:25.189445+0000","last_became_peered":"2026-02-22T19:52:25.189445+0000","last_unstale":"2026-02-22T20:32:24.867534+0000","last_undegraded":"2026-02-22T20:32:24.867534+0000","last_fullsized":"2026-02-22T20:32:24.867534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:10:12.693851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075128700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'72","reported_seq":918,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.867235+0000","last_change":"2026-02-22T19:52:23.183055+0000","last_active":"2026-02-22T20:32:24.867235+0000","last_peered":"2026-02-22T20:32:24.867235+0000","last_clean":"2026-02-22T20:32:24.867235+0000","last_became_active":"2026-02-22T19:52:23.182893+0000","last_became_peered":"2026-02-22T19:52:23.182893+0000","last_unstale":"2026-02-22T20:32:24.867235+0000","last_undegraded":"2026-02-22T20:32:24.867235+0000","last_fullsized":"2026-02-22T20:32:24.867235+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_clean_scrub_stamp":"2026-02-22T19:52:22.177365+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:20:38.106341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706153,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1548620,"kb_used_data":323004,"kb_used_omap":231,"kb_used_meta":1225368,"kb_avail":92823220,"statfs":{"total":96636764160,"available":95050977280,"internally_reserved":0,"allocated":330756096,"data_stored":327616113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236738,"internal_metadata":1254777662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706155,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454044,"kb_used_data":355276,"kb_used_omap":16,"kb_used_meta":1098735,"kb_avail":92917796,"statfs":{"total":96636764160,"available":95147823104,"internally_reserved":0,"allocated":363802624,"data_stored":360648327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17079,"internal_metadata":1125104969},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1389604,"kb_used_data":292564,"kb_used_omap":265,"kb_used_meta":1096758,"kb_avail":92982236,"statfs":{"total":96636764160,"available":95213809664,"internally_reserved":0,"allocated":299585536,"data_stored":296444351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":272202,"internal_metadata":1123080374},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:32:51.104 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:32:51.256 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T20:32:51.257 INFO:teuthology.orchestra.run.trial066.stderr:dumped all 2026-02-22T20:32:51.267 INFO:teuthology.orchestra.run.trial066.stdout:{"pg_ready":true,"pg_map":{"version":1627,"stamp":"2026-02-22T20:32:49.926899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4392268,"kb_used_data":970844,"kb_used_omap":513,"kb_used_meta":3420862,"kb_avail":278723252,"statfs":{"total":289910292480,"available":285412610048,"internally_reserved":0,"allocated":994144256,"data_stored":984708791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":526019,"internal_metadata":3502963005},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-725831795,"num_objects":-188,"num_object_clones":0,"num_object_copies":-376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-188,"num_whiteouts":0,"num_read":-1990270,"num_read_kb":-78973599,"num_write":-1893668,"num_write_kb":-71753999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.103568"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.868606+0000","last_change":"2026-02-22T19:52:27.193486+0000","last_active":"2026-02-22T20:32:24.868606+0000","last_peered":"2026-02-22T20:32:24.868606+0000","last_clean":"2026-02-22T20:32:24.868606+0000","last_became_active":"2026-02-22T19:52:25.190488+0000","last_became_peered":"2026-02-22T19:52:25.190488+0000","last_unstale":"2026-02-22T20:32:24.868606+0000","last_undegraded":"2026-02-22T20:32:24.868606+0000","last_fullsized":"2026-02-22T20:32:24.868606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:38:23.678969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091298699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.868760+0000","last_change":"2026-02-22T19:52:27.192944+0000","last_active":"2026-02-22T20:32:24.868760+0000","last_peered":"2026-02-22T20:32:24.868760+0000","last_clean":"2026-02-22T20:32:24.868760+0000","last_became_active":"2026-02-22T19:52:25.190482+0000","last_became_peered":"2026-02-22T19:52:25.190482+0000","last_unstale":"2026-02-22T20:32:24.868760+0000","last_undegraded":"2026-02-22T20:32:24.868760+0000","last_fullsized":"2026-02-22T20:32:24.868760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:29:08.459932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037698099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.868667+0000","last_change":"2026-02-22T19:52:27.193158+0000","last_active":"2026-02-22T20:32:24.868667+0000","last_peered":"2026-02-22T20:32:24.868667+0000","last_clean":"2026-02-22T20:32:24.868667+0000","last_became_active":"2026-02-22T19:52:25.189560+0000","last_became_peered":"2026-02-22T19:52:25.189560+0000","last_unstale":"2026-02-22T20:32:24.868667+0000","last_undegraded":"2026-02-22T20:32:24.868667+0000","last_fullsized":"2026-02-22T20:32:24.868667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:07:26.737908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059989500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.866993+0000","last_change":"2026-02-22T19:52:27.192933+0000","last_active":"2026-02-22T20:32:24.866993+0000","last_peered":"2026-02-22T20:32:24.866993+0000","last_clean":"2026-02-22T20:32:24.866993+0000","last_became_active":"2026-02-22T19:52:25.189562+0000","last_became_peered":"2026-02-22T19:52:25.189562+0000","last_unstale":"2026-02-22T20:32:24.866993+0000","last_undegraded":"2026-02-22T20:32:24.866993+0000","last_fullsized":"2026-02-22T20:32:24.866993+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:33:55.622679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038538599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.882362+0000","last_change":"2026-02-22T19:52:27.190720+0000","last_active":"2026-02-22T20:32:24.882362+0000","last_peered":"2026-02-22T20:32:24.882362+0000","last_clean":"2026-02-22T20:32:24.882362+0000","last_became_active":"2026-02-22T19:52:25.189656+0000","last_became_peered":"2026-02-22T19:52:25.189656+0000","last_unstale":"2026-02-22T20:32:24.882362+0000","last_undegraded":"2026-02-22T20:32:24.882362+0000","last_fullsized":"2026-02-22T20:32:24.882362+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:44:53.529788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026044399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:25.109454+0000","last_change":"2026-02-22T19:52:27.516719+0000","last_active":"2026-02-22T20:32:25.109454+0000","last_peered":"2026-02-22T20:32:25.109454+0000","last_clean":"2026-02-22T20:32:25.109454+0000","last_became_active":"2026-02-22T19:52:25.189998+0000","last_became_peered":"2026-02-22T19:52:25.189998+0000","last_unstale":"2026-02-22T20:32:25.109454+0000","last_undegraded":"2026-02-22T20:32:25.109454+0000","last_fullsized":"2026-02-22T20:32:25.109454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:34:47.988806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026744600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:25.109725+0000","last_change":"2026-02-22T19:52:27.516715+0000","last_active":"2026-02-22T20:32:25.109725+0000","last_peered":"2026-02-22T20:32:25.109725+0000","last_clean":"2026-02-22T20:32:25.109725+0000","last_became_active":"2026-02-22T19:52:25.189993+0000","last_became_peered":"2026-02-22T19:52:25.189993+0000","last_unstale":"2026-02-22T20:32:25.109725+0000","last_undegraded":"2026-02-22T20:32:25.109725+0000","last_fullsized":"2026-02-22T20:32:25.109725+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:48:18.874126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027272700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.867534+0000","last_change":"2026-02-22T19:52:27.193332+0000","last_active":"2026-02-22T20:32:24.867534+0000","last_peered":"2026-02-22T20:32:24.867534+0000","last_clean":"2026-02-22T20:32:24.867534+0000","last_became_active":"2026-02-22T19:52:25.189445+0000","last_became_peered":"2026-02-22T19:52:25.189445+0000","last_unstale":"2026-02-22T20:32:24.867534+0000","last_undegraded":"2026-02-22T20:32:24.867534+0000","last_fullsized":"2026-02-22T20:32:24.867534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:10:12.693851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075128700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'72","reported_seq":918,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.867235+0000","last_change":"2026-02-22T19:52:23.183055+0000","last_active":"2026-02-22T20:32:24.867235+0000","last_peered":"2026-02-22T20:32:24.867235+0000","last_clean":"2026-02-22T20:32:24.867235+0000","last_became_active":"2026-02-22T19:52:23.182893+0000","last_became_peered":"2026-02-22T19:52:23.182893+0000","last_unstale":"2026-02-22T20:32:24.867235+0000","last_undegraded":"2026-02-22T20:32:24.867235+0000","last_fullsized":"2026-02-22T20:32:24.867235+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_clean_scrub_stamp":"2026-02-22T19:52:22.177365+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:20:38.106341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706153,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1548620,"kb_used_data":323004,"kb_used_omap":231,"kb_used_meta":1225368,"kb_avail":92823220,"statfs":{"total":96636764160,"available":95050977280,"internally_reserved":0,"allocated":330756096,"data_stored":327616113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236738,"internal_metadata":1254777662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706155,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454044,"kb_used_data":355276,"kb_used_omap":16,"kb_used_meta":1098735,"kb_avail":92917796,"statfs":{"total":96636764160,"available":95147823104,"internally_reserved":0,"allocated":363802624,"data_stored":360648327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17079,"internal_metadata":1125104969},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1389604,"kb_used_data":292564,"kb_used_omap":265,"kb_used_meta":1096758,"kb_avail":92982236,"statfs":{"total":96636764160,"available":95213809664,"internally_reserved":0,"allocated":299585536,"data_stored":296444351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":272202,"internal_metadata":1123080374},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:32:51.268 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2026-02-22T20:32:51.268 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:32:51.422 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T20:32:51.423 INFO:teuthology.orchestra.run.trial066.stderr:dumped all 2026-02-22T20:32:51.432 INFO:teuthology.orchestra.run.trial066.stdout:{"pg_ready":true,"pg_map":{"version":1627,"stamp":"2026-02-22T20:32:49.926899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4392268,"kb_used_data":970844,"kb_used_omap":513,"kb_used_meta":3420862,"kb_avail":278723252,"statfs":{"total":289910292480,"available":285412610048,"internally_reserved":0,"allocated":994144256,"data_stored":984708791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":526019,"internal_metadata":3502963005},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-725831795,"num_objects":-188,"num_object_clones":0,"num_object_copies":-376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-188,"num_whiteouts":0,"num_read":-1990270,"num_read_kb":-78973599,"num_write":-1893668,"num_write_kb":-71753999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.103568"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.868606+0000","last_change":"2026-02-22T19:52:27.193486+0000","last_active":"2026-02-22T20:32:24.868606+0000","last_peered":"2026-02-22T20:32:24.868606+0000","last_clean":"2026-02-22T20:32:24.868606+0000","last_became_active":"2026-02-22T19:52:25.190488+0000","last_became_peered":"2026-02-22T19:52:25.190488+0000","last_unstale":"2026-02-22T20:32:24.868606+0000","last_undegraded":"2026-02-22T20:32:24.868606+0000","last_fullsized":"2026-02-22T20:32:24.868606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:38:23.678969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091298699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.868760+0000","last_change":"2026-02-22T19:52:27.192944+0000","last_active":"2026-02-22T20:32:24.868760+0000","last_peered":"2026-02-22T20:32:24.868760+0000","last_clean":"2026-02-22T20:32:24.868760+0000","last_became_active":"2026-02-22T19:52:25.190482+0000","last_became_peered":"2026-02-22T19:52:25.190482+0000","last_unstale":"2026-02-22T20:32:24.868760+0000","last_undegraded":"2026-02-22T20:32:24.868760+0000","last_fullsized":"2026-02-22T20:32:24.868760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:29:08.459932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037698099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.868667+0000","last_change":"2026-02-22T19:52:27.193158+0000","last_active":"2026-02-22T20:32:24.868667+0000","last_peered":"2026-02-22T20:32:24.868667+0000","last_clean":"2026-02-22T20:32:24.868667+0000","last_became_active":"2026-02-22T19:52:25.189560+0000","last_became_peered":"2026-02-22T19:52:25.189560+0000","last_unstale":"2026-02-22T20:32:24.868667+0000","last_undegraded":"2026-02-22T20:32:24.868667+0000","last_fullsized":"2026-02-22T20:32:24.868667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:07:26.737908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059989500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.866993+0000","last_change":"2026-02-22T19:52:27.192933+0000","last_active":"2026-02-22T20:32:24.866993+0000","last_peered":"2026-02-22T20:32:24.866993+0000","last_clean":"2026-02-22T20:32:24.866993+0000","last_became_active":"2026-02-22T19:52:25.189562+0000","last_became_peered":"2026-02-22T19:52:25.189562+0000","last_unstale":"2026-02-22T20:32:24.866993+0000","last_undegraded":"2026-02-22T20:32:24.866993+0000","last_fullsized":"2026-02-22T20:32:24.866993+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:33:55.622679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038538599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.882362+0000","last_change":"2026-02-22T19:52:27.190720+0000","last_active":"2026-02-22T20:32:24.882362+0000","last_peered":"2026-02-22T20:32:24.882362+0000","last_clean":"2026-02-22T20:32:24.882362+0000","last_became_active":"2026-02-22T19:52:25.189656+0000","last_became_peered":"2026-02-22T19:52:25.189656+0000","last_unstale":"2026-02-22T20:32:24.882362+0000","last_undegraded":"2026-02-22T20:32:24.882362+0000","last_fullsized":"2026-02-22T20:32:24.882362+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:44:53.529788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026044399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:25.109454+0000","last_change":"2026-02-22T19:52:27.516719+0000","last_active":"2026-02-22T20:32:25.109454+0000","last_peered":"2026-02-22T20:32:25.109454+0000","last_clean":"2026-02-22T20:32:25.109454+0000","last_became_active":"2026-02-22T19:52:25.189998+0000","last_became_peered":"2026-02-22T19:52:25.189998+0000","last_unstale":"2026-02-22T20:32:25.109454+0000","last_undegraded":"2026-02-22T20:32:25.109454+0000","last_fullsized":"2026-02-22T20:32:25.109454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:34:47.988806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026744600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:25.109725+0000","last_change":"2026-02-22T19:52:27.516715+0000","last_active":"2026-02-22T20:32:25.109725+0000","last_peered":"2026-02-22T20:32:25.109725+0000","last_clean":"2026-02-22T20:32:25.109725+0000","last_became_active":"2026-02-22T19:52:25.189993+0000","last_became_peered":"2026-02-22T19:52:25.189993+0000","last_unstale":"2026-02-22T20:32:25.109725+0000","last_undegraded":"2026-02-22T20:32:25.109725+0000","last_fullsized":"2026-02-22T20:32:25.109725+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:48:18.874126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027272700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.867534+0000","last_change":"2026-02-22T19:52:27.193332+0000","last_active":"2026-02-22T20:32:24.867534+0000","last_peered":"2026-02-22T20:32:24.867534+0000","last_clean":"2026-02-22T20:32:24.867534+0000","last_became_active":"2026-02-22T19:52:25.189445+0000","last_became_peered":"2026-02-22T19:52:25.189445+0000","last_unstale":"2026-02-22T20:32:24.867534+0000","last_undegraded":"2026-02-22T20:32:24.867534+0000","last_fullsized":"2026-02-22T20:32:24.867534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:10:12.693851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075128700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'72","reported_seq":918,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.867235+0000","last_change":"2026-02-22T19:52:23.183055+0000","last_active":"2026-02-22T20:32:24.867235+0000","last_peered":"2026-02-22T20:32:24.867235+0000","last_clean":"2026-02-22T20:32:24.867235+0000","last_became_active":"2026-02-22T19:52:23.182893+0000","last_became_peered":"2026-02-22T19:52:23.182893+0000","last_unstale":"2026-02-22T20:32:24.867235+0000","last_undegraded":"2026-02-22T20:32:24.867235+0000","last_fullsized":"2026-02-22T20:32:24.867235+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_clean_scrub_stamp":"2026-02-22T19:52:22.177365+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:20:38.106341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706153,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1548620,"kb_used_data":323004,"kb_used_omap":231,"kb_used_meta":1225368,"kb_avail":92823220,"statfs":{"total":96636764160,"available":95050977280,"internally_reserved":0,"allocated":330756096,"data_stored":327616113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236738,"internal_metadata":1254777662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706155,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454044,"kb_used_data":355276,"kb_used_omap":16,"kb_used_meta":1098735,"kb_avail":92917796,"statfs":{"total":96636764160,"available":95147823104,"internally_reserved":0,"allocated":363802624,"data_stored":360648327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17079,"internal_metadata":1125104969},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1389604,"kb_used_data":292564,"kb_used_omap":265,"kb_used_meta":1096758,"kb_avail":92982236,"statfs":{"total":96636764160,"available":95213809664,"internally_reserved":0,"allocated":299585536,"data_stored":296444351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":272202,"internal_metadata":1123080374},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:32:51.433 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-22T20:32:51.591 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T20:32:51.591 INFO:teuthology.orchestra.run.trial066.stdout:{"epoch":414,"fsid":"01607384-c85d-4035-8885-8f357ae44922","created":"2026-02-22T19:52:09.703535+0000","modified":"2026-02-22T20:32:49.917841+0000","last_up_change":"2026-02-22T19:52:20.171914+0000","last_in_change":"2026-02-22T19:52:10.264244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:52:21.436241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:52:23.837160+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ec0acef-b32e-4a62-a217-f9fbf183a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6808","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6809","nonce":3981027600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6810","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6811","nonce":3981027600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6814","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6815","nonce":3981027600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6812","nonce":3981027600},{"type":"v1","addr":"10.20.193.66:6813","nonce":3981027600}]},"public_addr":"10.20.193.66:6809/3981027600","cluster_addr":"10.20.193.66:6811/3981027600","heartbeat_back_addr":"10.20.193.66:6815/3981027600","heartbeat_front_addr":"10.20.193.66:6813/3981027600","state":["exists","up"]},{"osd":1,"uuid":"f352b79a-db0e-4c15-b625-23a8ab9a50b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6816","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6817","nonce":747693529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6818","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6819","nonce":747693529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6822","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6823","nonce":747693529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6820","nonce":747693529},{"type":"v1","addr":"10.20.193.66:6821","nonce":747693529}]},"public_addr":"10.20.193.66:6817/747693529","cluster_addr":"10.20.193.66:6819/747693529","heartbeat_back_addr":"10.20.193.66:6823/747693529","heartbeat_front_addr":"10.20.193.66:6821/747693529","state":["exists","up"]},{"osd":2,"uuid":"66d84059-d5b1-41a6-9820-620d0015f5bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6800","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6801","nonce":1448960510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6802","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6803","nonce":1448960510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6806","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6807","nonce":1448960510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.66:6804","nonce":1448960510},{"type":"v1","addr":"10.20.193.66:6805","nonce":1448960510}]},"public_addr":"10.20.193.66:6801/1448960510","cluster_addr":"10.20.193.66:6803/1448960510","heartbeat_back_addr":"10.20.193.66:6807/1448960510","heartbeat_front_addr":"10.20.193.66:6805/1448960510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.166499+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.348855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:18.140264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.66:0/1439552288":"2026-02-22T21:31:33.027775+0000","10.20.193.66:0/2623132544":"2026-02-22T21:30:52.392070+0000","10.20.193.66:0/2146391730":"2026-02-22T21:28:49.195610+0000","10.20.193.66:0/2460418969":"2026-02-22T21:28:05.419777+0000","10.20.193.66:0/915091148":"2026-02-22T21:27:28.190646+0000","10.20.193.66:0/2294137699":"2026-02-22T21:17:43.869473+0000","10.20.193.66:0/2715057279":"2026-02-22T21:17:25.842109+0000","10.20.193.66:0/2643151267":"2026-02-22T21:24:50.416418+0000","10.20.193.66:0/3758262779":"2026-02-22T21:20:59.951642+0000","10.20.193.66:0/545905834":"2026-02-22T21:23:23.871038+0000","10.20.193.66:0/14337587":"2026-02-22T21:16:06.616391+0000","10.20.193.66:0/572601501":"2026-02-22T21:16:33.073691+0000","10.20.193.66:0/3875354147":"2026-02-22T21:22:00.631192+0000","10.20.193.66:0/2191250487":"2026-02-22T21:17:13.957958+0000","10.20.193.66:0/2549407153":"2026-02-22T21:21:12.043458+0000","10.20.193.66:0/2932709274":"2026-02-22T21:32:09.842440+0000","10.20.193.66:0/3869108171":"2026-02-22T21:15:49.027505+0000","10.20.193.66:0/1382980494":"2026-02-22T21:15:26.651350+0000","10.20.193.66:0/908922920":"2026-02-22T21:18:17.079783+0000","10.20.193.66:0/330697421":"2026-02-22T21:22:43.315782+0000","10.20.193.66:0/3897641413":"2026-02-22T21:15:10.568155+0000","10.20.193.66:0/3744708506":"2026-02-22T21:04:14.607749+0000","10.20.193.66:0/3652337633":"2026-02-22T21:15:41.072941+0000","10.20.193.66:0/823628684":"2026-02-22T20:54:21.535895+0000","10.20.193.66:0/644613503":"2026-02-22T21:19:18.819621+0000","10.20.193.66:0/3970356081":"2026-02-22T21:20:21.396319+0000","10.20.193.66:0/2433764788":"2026-02-22T21:30:14.162075+0000","10.20.193.66:0/2888758782":"2026-02-22T21:21:44.765641+0000","10.20.193.66:0/2181045686":"2026-02-22T20:58:01.281386+0000","10.20.193.66:0/193621534":"2026-02-22T20:56:02.018243+0000","10.20.193.66:0/2881156504":"2026-02-22T21:05:42.717975+0000","10.20.193.66:0/1160234648":"2026-02-22T21:02:40.790842+0000","10.20.193.66:0/4093091357":"2026-02-22T21:15:55.259582+0000","10.20.193.66:0/1610771706":"2026-02-22T21:16:40.269538+0000","10.20.193.66:0/2049798330":"2026-02-22T20:59:38.460579+0000","10.20.193.66:0/2147757468":"2026-02-22T21:17:50.726677+0000","10.20.193.66:0/537499845":"2026-02-22T21:08:10.558342+0000","10.20.193.66:0/481566577":"2026-02-22T21:32:46.599942+0000","10.20.193.66:0/2703692210":"2026-02-22T21:15:16.649800+0000","10.20.193.66:0/3274338470":"2026-02-22T21:01:43.449521+0000","10.20.193.66:0/871698449":"2026-02-22T21:10:58.384809+0000","10.20.193.66:0/2237094552":"2026-02-22T21:16:21.465265+0000","10.20.193.66:0/3519486459":"2026-02-22T21:18:04.339058+0000","10.20.193.66:0/206327901":"2026-02-22T21:23:46.082844+0000","10.20.193.66:0/2711361783":"2026-02-22T21:24:19.578221+0000","10.20.193.66:0/4223524786":"2026-02-22T21:11:06.289235+0000","10.20.193.66:0/3448950050":"2026-02-22T21:20:09.792311+0000","10.20.193.66:0/3215985592":"2026-02-22T21:16:45.971367+0000","10.20.193.66:0/924313873":"2026-02-22T21:00:40.539970+0000","10.20.193.66:0/1523445817":"2026-02-22T21:19:48.985415+0000","10.20.193.66:0/1347477156":"2026-02-22T21:18:31.874578+0000","10.20.193.66:0/1271864539":"2026-02-22T21:18:38.736482+0000","10.20.193.66:0/2686679218":"2026-02-22T21:19:35.346114+0000","10.20.193.66:0/383657073":"2026-02-22T21:19:26.057664+0000","10.20.193.66:0/3088572450":"2026-02-22T21:21:55.746688+0000","10.20.193.66:0/2634832931":"2026-02-22T20:56:57.174187+0000","10.20.193.66:0/2267707729":"2026-02-22T21:20:01.686904+0000","10.20.193.66:0/2565970096":"2026-02-22T21:21:23.533123+0000","10.20.193.66:0/2530164529":"2026-02-22T21:29:31.454045+0000","10.20.193.66:0/3618101613":"2026-02-22T21:21:28.404565+0000","10.20.193.66:0/1791996801":"2026-02-22T21:21:37.467481+0000","10.20.193.66:0/3417752938":"2026-02-22T21:24:10.485736+0000","10.20.193.66:0/1512506104":"2026-02-22T21:22:57.878706+0000","10.20.193.66:0/751584927":"2026-02-22T21:23:05.910835+0000","10.20.193.66:0/3159102929":"2026-02-22T21:26:31.318533+0000","10.20.193.66:0/1837208838":"2026-02-22T21:23:35.334520+0000","10.20.193.66:0/514173770":"2026-02-22T21:24:01.581341+0000","10.20.193.66:0/869069628":"2026-02-22T21:13:27.135085+0000","10.20.193.66:0/2668172433":"2026-02-22T21:18:50.317185+0000","10.20.193.66:0/1999400433":"2026-02-22T21:24:35.538809+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T20:32:52.604 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2026-02-22T20:32:52.604 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:32:52.681 INFO:teuthology.orchestra.run.trial066.stdout:{ 2026-02-22T20:32:52.682 INFO:teuthology.orchestra.run.trial066.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:32:52.682 INFO:teuthology.orchestra.run.trial066.stdout:} 2026-02-22T20:32:52.691 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2026-02-22T20:32:52.845 INFO:teuthology.orchestra.run.trial066.stderr:instructed osd(s) 0 to deep-scrub 2026-02-22T20:32:52.856 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2026-02-22T20:32:52.857 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:32:52.938 INFO:teuthology.orchestra.run.trial066.stdout:{ 2026-02-22T20:32:52.938 INFO:teuthology.orchestra.run.trial066.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T20:32:52.938 INFO:teuthology.orchestra.run.trial066.stdout:} 2026-02-22T20:32:52.948 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2026-02-22T20:32:53.104 INFO:teuthology.orchestra.run.trial066.stderr:instructed osd(s) 1 to deep-scrub 2026-02-22T20:32:53.115 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2026-02-22T20:32:53.115 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:32:53.197 INFO:teuthology.orchestra.run.trial066.stdout:{ 2026-02-22T20:32:53.197 INFO:teuthology.orchestra.run.trial066.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T20:32:53.197 INFO:teuthology.orchestra.run.trial066.stdout:} 2026-02-22T20:32:53.206 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2026-02-22T20:32:53.483 INFO:teuthology.orchestra.run.trial066.stderr:instructed osd(s) 2 to deep-scrub 2026-02-22T20:32:53.495 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:32:53.648 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T20:32:53.649 INFO:teuthology.orchestra.run.trial066.stderr:dumped all 2026-02-22T20:32:53.659 INFO:teuthology.orchestra.run.trial066.stdout:{"pg_ready":true,"pg_map":{"version":1628,"stamp":"2026-02-22T20:32:51.824411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3594420,"kb_used_data":365436,"kb_used_omap":513,"kb_used_meta":3228414,"kb_avail":279521100,"statfs":{"total":289910292480,"available":286229606400,"internally_reserved":0,"allocated":374206464,"data_stored":365789070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":526019,"internal_metadata":3305896253},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-725831795,"num_objects":-188,"num_object_clones":0,"num_object_copies":-376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-188,"num_whiteouts":0,"num_read":-1990276,"num_read_kb":-78973604,"num_write":-1893669,"num_write_kb":-71754000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000756"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":832,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.922048+0000","last_change":"2026-02-22T19:52:27.193486+0000","last_active":"2026-02-22T20:32:49.922048+0000","last_peered":"2026-02-22T20:32:49.922048+0000","last_clean":"2026-02-22T20:32:49.922048+0000","last_became_active":"2026-02-22T19:52:25.190488+0000","last_became_peered":"2026-02-22T19:52:25.190488+0000","last_unstale":"2026-02-22T20:32:49.922048+0000","last_undegraded":"2026-02-22T20:32:49.922048+0000","last_fullsized":"2026-02-22T20:32:49.922048+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:38:23.678969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091298699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":832,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.922043+0000","last_change":"2026-02-22T19:52:27.192944+0000","last_active":"2026-02-22T20:32:49.922043+0000","last_peered":"2026-02-22T20:32:49.922043+0000","last_clean":"2026-02-22T20:32:49.922043+0000","last_became_active":"2026-02-22T19:52:25.190482+0000","last_became_peered":"2026-02-22T19:52:25.190482+0000","last_unstale":"2026-02-22T20:32:49.922043+0000","last_undegraded":"2026-02-22T20:32:49.922043+0000","last_fullsized":"2026-02-22T20:32:49.922043+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:29:08.459932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037698099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":833,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.921977+0000","last_change":"2026-02-22T19:52:27.193158+0000","last_active":"2026-02-22T20:32:49.921977+0000","last_peered":"2026-02-22T20:32:49.921977+0000","last_clean":"2026-02-22T20:32:49.921977+0000","last_became_active":"2026-02-22T19:52:25.189560+0000","last_became_peered":"2026-02-22T19:52:25.189560+0000","last_unstale":"2026-02-22T20:32:49.921977+0000","last_undegraded":"2026-02-22T20:32:49.921977+0000","last_fullsized":"2026-02-22T20:32:49.921977+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:07:26.737908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059989500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":833,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.921967+0000","last_change":"2026-02-22T19:52:27.192933+0000","last_active":"2026-02-22T20:32:49.921967+0000","last_peered":"2026-02-22T20:32:49.921967+0000","last_clean":"2026-02-22T20:32:49.921967+0000","last_became_active":"2026-02-22T19:52:25.189562+0000","last_became_peered":"2026-02-22T19:52:25.189562+0000","last_unstale":"2026-02-22T20:32:49.921967+0000","last_undegraded":"2026-02-22T20:32:49.921967+0000","last_fullsized":"2026-02-22T20:32:49.921967+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:33:55.622679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038538599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":829,"reported_epoch":411,"state":"active+clean","last_fresh":"2026-02-22T20:32:24.882362+0000","last_change":"2026-02-22T19:52:27.190720+0000","last_active":"2026-02-22T20:32:24.882362+0000","last_peered":"2026-02-22T20:32:24.882362+0000","last_clean":"2026-02-22T20:32:24.882362+0000","last_became_active":"2026-02-22T19:52:25.189656+0000","last_became_peered":"2026-02-22T19:52:25.189656+0000","last_unstale":"2026-02-22T20:32:24.882362+0000","last_undegraded":"2026-02-22T20:32:24.882362+0000","last_fullsized":"2026-02-22T20:32:24.882362+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:44:53.529788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026044399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":833,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.939259+0000","last_change":"2026-02-22T19:52:27.516719+0000","last_active":"2026-02-22T20:32:49.939259+0000","last_peered":"2026-02-22T20:32:49.939259+0000","last_clean":"2026-02-22T20:32:49.939259+0000","last_became_active":"2026-02-22T19:52:25.189998+0000","last_became_peered":"2026-02-22T19:52:25.189998+0000","last_unstale":"2026-02-22T20:32:49.939259+0000","last_undegraded":"2026-02-22T20:32:49.939259+0000","last_fullsized":"2026-02-22T20:32:49.939259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:34:47.988806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026744600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":834,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.940551+0000","last_change":"2026-02-22T19:52:27.516715+0000","last_active":"2026-02-22T20:32:49.940551+0000","last_peered":"2026-02-22T20:32:49.940551+0000","last_clean":"2026-02-22T20:32:49.940551+0000","last_became_active":"2026-02-22T19:52:25.189993+0000","last_became_peered":"2026-02-22T19:52:25.189993+0000","last_unstale":"2026-02-22T20:32:49.940551+0000","last_undegraded":"2026-02-22T20:32:49.940551+0000","last_fullsized":"2026-02-22T20:32:49.940551+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:48:18.874126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027272700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":834,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.929539+0000","last_change":"2026-02-22T19:52:27.193332+0000","last_active":"2026-02-22T20:32:49.929539+0000","last_peered":"2026-02-22T20:32:49.929539+0000","last_clean":"2026-02-22T20:32:49.929539+0000","last_became_active":"2026-02-22T19:52:25.189445+0000","last_became_peered":"2026-02-22T19:52:25.189445+0000","last_unstale":"2026-02-22T20:32:49.929539+0000","last_undegraded":"2026-02-22T20:32:49.929539+0000","last_fullsized":"2026-02-22T20:32:49.929539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:24.181547+0000","last_clean_scrub_stamp":"2026-02-22T19:52:24.181547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:10:12.693851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075128700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'72","reported_seq":923,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.929775+0000","last_change":"2026-02-22T19:52:23.183055+0000","last_active":"2026-02-22T20:32:49.929775+0000","last_peered":"2026-02-22T20:32:49.929775+0000","last_clean":"2026-02-22T20:32:49.929775+0000","last_became_active":"2026-02-22T19:52:23.182893+0000","last_became_peered":"2026-02-22T19:52:23.182893+0000","last_unstale":"2026-02-22T20:32:49.929775+0000","last_undegraded":"2026-02-22T20:32:49.929775+0000","last_fullsized":"2026-02-22T20:32:49.929775+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:22.177365+0000","last_clean_scrub_stamp":"2026-02-22T19:52:22.177365+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:20:38.106341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706154,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1038556,"kb_used_data":5384,"kb_used_omap":231,"kb_used_meta":1032920,"kb_avail":93333284,"statfs":{"total":96636764160,"available":95573282816,"internally_reserved":0,"allocated":5513216,"data_stored":2872421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236738,"internal_metadata":1057710910},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706155,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454044,"kb_used_data":355276,"kb_used_omap":16,"kb_used_meta":1098735,"kb_avail":92917796,"statfs":{"total":96636764160,"available":95147823104,"internally_reserved":0,"allocated":363802624,"data_stored":360648327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17079,"internal_metadata":1125104969},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738858,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1101820,"kb_used_data":4776,"kb_used_omap":265,"kb_used_meta":1096758,"kb_avail":93270020,"statfs":{"total":96636764160,"available":95508500480,"internally_reserved":0,"allocated":4890624,"data_stored":2268322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":272202,"internal_metadata":1123080374},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:32:53.661 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-22T19:52:24.181547+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=24, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=32, tm_sec=51, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:32:53.661 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-22T19:52:24.181547+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=24, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=32, tm_sec=51, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:32:53.661 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-22T19:52:24.181547+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=24, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=32, tm_sec=51, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:32:53.661 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-22T19:52:24.181547+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=24, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=32, tm_sec=51, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:32:53.661 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-22T19:52:24.181547+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=24, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=32, tm_sec=51, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:32:53.661 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-22T19:52:24.181547+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=24, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=32, tm_sec=51, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:32:53.661 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-22T19:52:24.181547+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=24, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=32, tm_sec=51, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:32:53.661 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-22T19:52:24.181547+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=24, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=32, tm_sec=51, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:32:53.662 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-22T19:52:22.177365+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=22, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=32, tm_sec=51, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:32:53.662 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-22T20:33:13.663 DEBUG:teuthology.orchestra.run.trial066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:33:13.815 INFO:teuthology.orchestra.run.trial066.stdout: 2026-02-22T20:33:13.815 INFO:teuthology.orchestra.run.trial066.stderr:dumped all 2026-02-22T20:33:13.825 INFO:teuthology.orchestra.run.trial066.stdout:{"pg_ready":true,"pg_map":{"version":1638,"stamp":"2026-02-22T20:33:11.826197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3242368,"kb_used_data":15568,"kb_used_omap":513,"kb_used_meta":3226238,"kb_avail":279873152,"statfs":{"total":289910292480,"available":286590107648,"internally_reserved":0,"allocated":15941632,"data_stored":8026876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":526019,"internal_metadata":3303668029},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000984"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":841,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:56.181940+0000","last_change":"2026-02-22T20:32:56.181965+0000","last_active":"2026-02-22T20:32:56.181940+0000","last_peered":"2026-02-22T20:32:56.181940+0000","last_clean":"2026-02-22T20:32:56.181940+0000","last_became_active":"2026-02-22T19:52:25.190488+0000","last_became_peered":"2026-02-22T19:52:25.190488+0000","last_unstale":"2026-02-22T20:32:56.181940+0000","last_undegraded":"2026-02-22T20:32:56.181940+0000","last_fullsized":"2026-02-22T20:32:56.181940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:32:56.181854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:32:56.181854+0000","last_clean_scrub_stamp":"2026-02-22T20:32:56.181854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:04:29.212125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091298699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":841,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:58.159739+0000","last_change":"2026-02-22T20:32:58.159778+0000","last_active":"2026-02-22T20:32:58.159739+0000","last_peered":"2026-02-22T20:32:58.159739+0000","last_clean":"2026-02-22T20:32:58.159739+0000","last_became_active":"2026-02-22T19:52:25.190482+0000","last_became_peered":"2026-02-22T19:52:25.190482+0000","last_unstale":"2026-02-22T20:32:58.159739+0000","last_undegraded":"2026-02-22T20:32:58.159739+0000","last_fullsized":"2026-02-22T20:32:58.159739+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:32:58.157381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:32:58.157381+0000","last_clean_scrub_stamp":"2026-02-22T20:32:58.157381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:24:21.851157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037698099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":842,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:55.164590+0000","last_change":"2026-02-22T20:32:55.164606+0000","last_active":"2026-02-22T20:32:55.164590+0000","last_peered":"2026-02-22T20:32:55.164590+0000","last_clean":"2026-02-22T20:32:55.164590+0000","last_became_active":"2026-02-22T19:52:25.189560+0000","last_became_peered":"2026-02-22T19:52:25.189560+0000","last_unstale":"2026-02-22T20:32:55.164590+0000","last_undegraded":"2026-02-22T20:32:55.164590+0000","last_fullsized":"2026-02-22T20:32:55.164590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:32:55.164525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:32:55.164525+0000","last_clean_scrub_stamp":"2026-02-22T20:32:55.164525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:40:45.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059989500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":842,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:54.203755+0000","last_change":"2026-02-22T20:32:54.203793+0000","last_active":"2026-02-22T20:32:54.203755+0000","last_peered":"2026-02-22T20:32:54.203755+0000","last_clean":"2026-02-22T20:32:54.203755+0000","last_became_active":"2026-02-22T19:52:25.189562+0000","last_became_peered":"2026-02-22T19:52:25.189562+0000","last_unstale":"2026-02-22T20:32:54.203755+0000","last_undegraded":"2026-02-22T20:32:54.203755+0000","last_fullsized":"2026-02-22T20:32:54.203755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:32:54.203671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:32:54.203671+0000","last_clean_scrub_stamp":"2026-02-22T20:32:54.203671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:31:55.269378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038538599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"414'3","reported_seq":845,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:53.225778+0000","last_change":"2026-02-22T20:32:53.225796+0000","last_active":"2026-02-22T20:32:53.225778+0000","last_peered":"2026-02-22T20:32:53.225778+0000","last_clean":"2026-02-22T20:32:53.225778+0000","last_became_active":"2026-02-22T19:52:25.189656+0000","last_became_peered":"2026-02-22T19:52:25.189656+0000","last_unstale":"2026-02-22T20:32:53.225778+0000","last_undegraded":"2026-02-22T20:32:53.225778+0000","last_fullsized":"2026-02-22T20:32:53.225778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'3","last_scrub_stamp":"2026-02-22T20:32:53.225731+0000","last_deep_scrub":"414'3","last_deep_scrub_stamp":"2026-02-22T20:32:53.225731+0000","last_clean_scrub_stamp":"2026-02-22T20:32:53.225731+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:19:05.506368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026044399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":842,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:55.463325+0000","last_change":"2026-02-22T20:32:55.463345+0000","last_active":"2026-02-22T20:32:55.463325+0000","last_peered":"2026-02-22T20:32:55.463325+0000","last_clean":"2026-02-22T20:32:55.463325+0000","last_became_active":"2026-02-22T19:52:25.189998+0000","last_became_peered":"2026-02-22T19:52:25.189998+0000","last_unstale":"2026-02-22T20:32:55.463325+0000","last_undegraded":"2026-02-22T20:32:55.463325+0000","last_fullsized":"2026-02-22T20:32:55.463325+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:32:55.463255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:32:55.463255+0000","last_clean_scrub_stamp":"2026-02-22T20:32:55.463255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:05:15.568974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00026744600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":843,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:54.438461+0000","last_change":"2026-02-22T20:32:54.438484+0000","last_active":"2026-02-22T20:32:54.438461+0000","last_peered":"2026-02-22T20:32:54.438461+0000","last_clean":"2026-02-22T20:32:54.438461+0000","last_became_active":"2026-02-22T19:52:25.189993+0000","last_became_peered":"2026-02-22T19:52:25.189993+0000","last_unstale":"2026-02-22T20:32:54.438461+0000","last_undegraded":"2026-02-22T20:32:54.438461+0000","last_fullsized":"2026-02-22T20:32:54.438461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:32:54.438376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:32:54.438376+0000","last_clean_scrub_stamp":"2026-02-22T20:32:54.438376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:49:44.438680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00027272700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"414'2","reported_seq":844,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:57.203690+0000","last_change":"2026-02-22T20:32:57.203729+0000","last_active":"2026-02-22T20:32:57.203690+0000","last_peered":"2026-02-22T20:32:57.203690+0000","last_clean":"2026-02-22T20:32:57.203690+0000","last_became_active":"2026-02-22T19:52:25.189445+0000","last_became_peered":"2026-02-22T19:52:25.189445+0000","last_unstale":"2026-02-22T20:32:57.203690+0000","last_undegraded":"2026-02-22T20:32:57.203690+0000","last_fullsized":"2026-02-22T20:32:57.203690+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'2","last_scrub_stamp":"2026-02-22T20:32:57.203585+0000","last_deep_scrub":"414'2","last_deep_scrub_stamp":"2026-02-22T20:32:57.203585+0000","last_clean_scrub_stamp":"2026-02-22T20:32:57.203585+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:51:45.741324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075128700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"414'74","reported_seq":934,"reported_epoch":414,"state":"active+clean","last_fresh":"2026-02-22T20:32:53.186078+0000","last_change":"2026-02-22T20:32:53.186103+0000","last_active":"2026-02-22T20:32:53.186078+0000","last_peered":"2026-02-22T20:32:53.186078+0000","last_clean":"2026-02-22T20:32:53.186078+0000","last_became_active":"2026-02-22T19:52:23.182893+0000","last_became_peered":"2026-02-22T19:52:23.182893+0000","last_unstale":"2026-02-22T20:32:53.186078+0000","last_undegraded":"2026-02-22T20:32:53.186078+0000","last_fullsized":"2026-02-22T20:32:53.186078+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'74","last_scrub_stamp":"2026-02-22T20:32:53.186032+0000","last_deep_scrub":"414'74","last_deep_scrub_stamp":"2026-02-22T20:32:53.186032+0000","last_clean_scrub_stamp":"2026-02-22T20:32:53.186032+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:17:58.160487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706158,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1038552,"kb_used_data":5384,"kb_used_omap":231,"kb_used_meta":1032920,"kb_avail":93333288,"statfs":{"total":96636764160,"available":95573286912,"internally_reserved":0,"allocated":5513216,"data_stored":2872421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236738,"internal_metadata":1057710910},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706159,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1097816,"kb_used_data":5384,"kb_used_omap":16,"kb_used_meta":1092399,"kb_avail":93274024,"statfs":{"total":96636764160,"available":95512600576,"internally_reserved":0,"allocated":5513216,"data_stored":2872421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17079,"internal_metadata":1118616905},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738862,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1106000,"kb_used_data":4800,"kb_used_omap":265,"kb_used_meta":1100918,"kb_avail":93265840,"statfs":{"total":96636764160,"available":95504220160,"internally_reserved":0,"allocated":4915200,"data_stored":2282034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":272202,"internal_metadata":1127340214},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:33:13.827 DEBUG:teuthology.orchestra.run.trial066:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2026-02-22T20:33:13.984 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-22T20:33:13.985 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-22T20:33:13.985 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2026-02-22T20:33:13.985 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:33:16.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2026-02-22T20:33:20.088 INFO:tasks.ceph.cluster2.osd.0:Stopped 2026-02-22T20:33:20.088 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2026-02-22T20:33:20.088 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:33:22.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2026-02-22T20:33:22.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2026-02-22T20:33:26.190 INFO:tasks.ceph.cluster2.osd.1:Stopped 2026-02-22T20:33:26.190 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2026-02-22T20:33:26.190 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:33:27.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2026-02-22T20:33:27.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2026-02-22T20:33:32.291 INFO:tasks.ceph.cluster2.osd.2:Stopped 2026-02-22T20:33:32.292 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-22T20:33:32.292 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2026-02-22T20:33:32.292 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:33:32.329 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2026-02-22T20:33:32.329 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-22T20:33:32.329 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2026-02-22T20:33:32.329 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:33:32.347 INFO:tasks.ceph.cluster2.mon.a:Stopped 2026-02-22T20:33:32.347 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-22T20:33:32.347 DEBUG:teuthology.orchestra.run.trial066:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2026-02-22T20:33:32.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@trial066.front.sepia.ceph.com 2026-02-22T20:33:32.401 DEBUG:teuthology.orchestra.run.trial066:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2026-02-22T20:33:32.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2026-02-22T20:33:33.176 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@trial066.front.sepia.ceph.com 2026-02-22T20:33:33.176 DEBUG:teuthology.orchestra.run.trial066:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2026-02-22T20:33:33.909 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@trial066.front.sepia.ceph.com 2026-02-22T20:33:33.909 DEBUG:teuthology.orchestra.run.trial066:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2026-02-22T20:33:34.587 INFO:tasks.ceph:Archiving mon data... 2026-02-22T20:33:34.589 DEBUG:teuthology.misc:Transferring archived files from trial066:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972/data/cluster2.mon.a.tgz 2026-02-22T20:33:34.590 DEBUG:teuthology.orchestra.run.trial066:> mktemp 2026-02-22T20:33:34.594 INFO:teuthology.orchestra.run.trial066.stdout:/tmp/tmp.wkpyAH7VUA 2026-02-22T20:33:34.595 DEBUG:teuthology.orchestra.run.trial066:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.wkpyAH7VUA 2026-02-22T20:33:34.713 DEBUG:teuthology.orchestra.run.trial066:> sudo chmod 0666 /tmp/tmp.wkpyAH7VUA 2026-02-22T20:33:34.775 DEBUG:teuthology.orchestra.remote:trial066:/tmp/tmp.wkpyAH7VUA is 491KB 2026-02-22T20:33:34.827 DEBUG:teuthology.orchestra.run.trial066:> rm -fr /tmp/tmp.wkpyAH7VUA 2026-02-22T20:33:34.831 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-22T20:33:34.831 DEBUG:teuthology.orchestra.run.trial042:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T20:33:34.832 DEBUG:teuthology.orchestra.run.trial066:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T20:33:34.882 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-22T20:33:34.898 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2026-02-22T20:33:34.899 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:33:35.041 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T20:33:35.041 INFO:teuthology.orchestra.run.trial042.stderr:dumped all 2026-02-22T20:33:35.051 INFO:teuthology.orchestra.run.trial042.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2026-02-22T20:33:33.630463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2836680,"kb_used_data":9280,"kb_used_omap":185,"kb_used_meta":2827142,"kb_avail":280278840,"statfs":{"total":289910292480,"available":287005532160,"internally_reserved":0,"allocated":9502720,"data_stored":5557033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":190427,"internal_metadata":2894993445},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001110"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.587232+0000","last_change":"2026-02-22T19:51:48.467655+0000","last_active":"2026-02-22T20:32:49.587232+0000","last_peered":"2026-02-22T20:32:49.587232+0000","last_clean":"2026-02-22T20:32:49.587232+0000","last_became_active":"2026-02-22T19:51:46.461368+0000","last_became_peered":"2026-02-22T19:51:46.461368+0000","last_unstale":"2026-02-22T20:32:49.587232+0000","last_undegraded":"2026-02-22T20:32:49.587232+0000","last_fullsized":"2026-02-22T20:32:49.587232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:54:49.652425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059369800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.589453+0000","last_change":"2026-02-22T19:51:48.467546+0000","last_active":"2026-02-22T20:32:49.589453+0000","last_peered":"2026-02-22T20:32:49.589453+0000","last_clean":"2026-02-22T20:32:49.589453+0000","last_became_active":"2026-02-22T19:51:46.461805+0000","last_became_peered":"2026-02-22T19:51:46.461805+0000","last_unstale":"2026-02-22T20:32:49.589453+0000","last_undegraded":"2026-02-22T20:32:49.589453+0000","last_fullsized":"2026-02-22T20:32:49.589453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:40:08.170741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048713899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.588595+0000","last_change":"2026-02-22T19:51:48.467382+0000","last_active":"2026-02-22T20:32:49.588595+0000","last_peered":"2026-02-22T20:32:49.588595+0000","last_clean":"2026-02-22T20:32:49.588595+0000","last_became_active":"2026-02-22T19:51:46.461823+0000","last_became_peered":"2026-02-22T19:51:46.461823+0000","last_unstale":"2026-02-22T20:32:49.588595+0000","last_undegraded":"2026-02-22T20:32:49.588595+0000","last_fullsized":"2026-02-22T20:32:49.588595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:03:33.088966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000469977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":678,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.586071+0000","last_change":"2026-02-22T19:51:48.468007+0000","last_active":"2026-02-22T20:32:49.586071+0000","last_peered":"2026-02-22T20:32:49.586071+0000","last_clean":"2026-02-22T20:32:49.586071+0000","last_became_active":"2026-02-22T19:51:46.461360+0000","last_became_peered":"2026-02-22T19:51:46.461360+0000","last_unstale":"2026-02-22T20:32:49.586071+0000","last_undegraded":"2026-02-22T20:32:49.586071+0000","last_fullsized":"2026-02-22T20:32:49.586071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:35:55.755801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000927201,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":681,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.275302+0000","last_change":"2026-02-22T19:51:48.465651+0000","last_active":"2026-02-22T20:32:49.275302+0000","last_peered":"2026-02-22T20:32:49.275302+0000","last_clean":"2026-02-22T20:32:49.275302+0000","last_became_active":"2026-02-22T19:51:46.461880+0000","last_became_peered":"2026-02-22T19:51:46.461880+0000","last_unstale":"2026-02-22T20:32:49.275302+0000","last_undegraded":"2026-02-22T20:32:49.275302+0000","last_fullsized":"2026-02-22T20:32:49.275302+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:20:16.291609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000706108,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:50.367546+0000","last_change":"2026-02-22T19:51:48.562746+0000","last_active":"2026-02-22T20:32:50.367546+0000","last_peered":"2026-02-22T20:32:50.367546+0000","last_clean":"2026-02-22T20:32:50.367546+0000","last_became_active":"2026-02-22T19:51:46.461871+0000","last_became_peered":"2026-02-22T19:51:46.461871+0000","last_unstale":"2026-02-22T20:32:50.367546+0000","last_undegraded":"2026-02-22T20:32:50.367546+0000","last_fullsized":"2026-02-22T20:32:50.367546+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:23.013120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028334900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:50.368324+0000","last_change":"2026-02-22T19:51:48.562686+0000","last_active":"2026-02-22T20:32:50.368324+0000","last_peered":"2026-02-22T20:32:50.368324+0000","last_clean":"2026-02-22T20:32:50.368324+0000","last_became_active":"2026-02-22T19:51:46.461861+0000","last_became_peered":"2026-02-22T19:51:46.461861+0000","last_unstale":"2026-02-22T20:32:50.368324+0000","last_undegraded":"2026-02-22T20:32:50.368324+0000","last_fullsized":"2026-02-22T20:32:50.368324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:28:58.573469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026117799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.587120+0000","last_change":"2026-02-22T19:51:48.467890+0000","last_active":"2026-02-22T20:32:49.587120+0000","last_peered":"2026-02-22T20:32:49.587120+0000","last_clean":"2026-02-22T20:32:49.587120+0000","last_became_active":"2026-02-22T19:51:46.461352+0000","last_became_peered":"2026-02-22T19:51:46.461352+0000","last_unstale":"2026-02-22T20:32:49.587120+0000","last_undegraded":"2026-02-22T20:32:49.587120+0000","last_fullsized":"2026-02-22T20:32:49.587120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:01:13.797556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082900699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":767,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.583793+0000","last_change":"2026-02-22T19:51:45.322266+0000","last_active":"2026-02-22T20:32:49.583793+0000","last_peered":"2026-02-22T20:32:49.583793+0000","last_clean":"2026-02-22T20:32:49.583793+0000","last_became_active":"2026-02-22T19:51:45.321492+0000","last_became_peered":"2026-02-22T19:51:45.321492+0000","last_unstale":"2026-02-22T20:32:49.583793+0000","last_undegraded":"2026-02-22T20:32:49.583793+0000","last_fullsized":"2026-02-22T20:32:49.583793+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_clean_scrub_stamp":"2026-02-22T19:51:43.450496+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:22:35.569483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706168,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1230312,"kb_used_data":2704,"kb_used_omap":40,"kb_used_meta":1227543,"kb_avail":93141528,"statfs":{"total":96636764160,"available":95376924672,"internally_reserved":0,"allocated":2768896,"data_stored":1458753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41475,"internal_metadata":1257004541},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706168,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761328,"kb_used_data":3288,"kb_used_omap":41,"kb_used_meta":757974,"kb_avail":93610512,"statfs":{"total":96636764160,"available":95857164288,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42838,"internal_metadata":776165546},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738875,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":845040,"kb_used_data":3288,"kb_used_omap":103,"kb_used_meta":841624,"kb_avail":93526800,"statfs":{"total":96636764160,"available":95771443200,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106114,"internal_metadata":861823358},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:33:35.051 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:33:35.193 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T20:33:35.194 INFO:teuthology.orchestra.run.trial042.stderr:dumped all 2026-02-22T20:33:35.204 INFO:teuthology.orchestra.run.trial042.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2026-02-22T20:33:33.630463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2836680,"kb_used_data":9280,"kb_used_omap":185,"kb_used_meta":2827142,"kb_avail":280278840,"statfs":{"total":289910292480,"available":287005532160,"internally_reserved":0,"allocated":9502720,"data_stored":5557033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":190427,"internal_metadata":2894993445},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001110"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.587232+0000","last_change":"2026-02-22T19:51:48.467655+0000","last_active":"2026-02-22T20:32:49.587232+0000","last_peered":"2026-02-22T20:32:49.587232+0000","last_clean":"2026-02-22T20:32:49.587232+0000","last_became_active":"2026-02-22T19:51:46.461368+0000","last_became_peered":"2026-02-22T19:51:46.461368+0000","last_unstale":"2026-02-22T20:32:49.587232+0000","last_undegraded":"2026-02-22T20:32:49.587232+0000","last_fullsized":"2026-02-22T20:32:49.587232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:54:49.652425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059369800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.589453+0000","last_change":"2026-02-22T19:51:48.467546+0000","last_active":"2026-02-22T20:32:49.589453+0000","last_peered":"2026-02-22T20:32:49.589453+0000","last_clean":"2026-02-22T20:32:49.589453+0000","last_became_active":"2026-02-22T19:51:46.461805+0000","last_became_peered":"2026-02-22T19:51:46.461805+0000","last_unstale":"2026-02-22T20:32:49.589453+0000","last_undegraded":"2026-02-22T20:32:49.589453+0000","last_fullsized":"2026-02-22T20:32:49.589453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:40:08.170741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048713899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.588595+0000","last_change":"2026-02-22T19:51:48.467382+0000","last_active":"2026-02-22T20:32:49.588595+0000","last_peered":"2026-02-22T20:32:49.588595+0000","last_clean":"2026-02-22T20:32:49.588595+0000","last_became_active":"2026-02-22T19:51:46.461823+0000","last_became_peered":"2026-02-22T19:51:46.461823+0000","last_unstale":"2026-02-22T20:32:49.588595+0000","last_undegraded":"2026-02-22T20:32:49.588595+0000","last_fullsized":"2026-02-22T20:32:49.588595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:03:33.088966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000469977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":678,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.586071+0000","last_change":"2026-02-22T19:51:48.468007+0000","last_active":"2026-02-22T20:32:49.586071+0000","last_peered":"2026-02-22T20:32:49.586071+0000","last_clean":"2026-02-22T20:32:49.586071+0000","last_became_active":"2026-02-22T19:51:46.461360+0000","last_became_peered":"2026-02-22T19:51:46.461360+0000","last_unstale":"2026-02-22T20:32:49.586071+0000","last_undegraded":"2026-02-22T20:32:49.586071+0000","last_fullsized":"2026-02-22T20:32:49.586071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:35:55.755801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000927201,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":681,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.275302+0000","last_change":"2026-02-22T19:51:48.465651+0000","last_active":"2026-02-22T20:32:49.275302+0000","last_peered":"2026-02-22T20:32:49.275302+0000","last_clean":"2026-02-22T20:32:49.275302+0000","last_became_active":"2026-02-22T19:51:46.461880+0000","last_became_peered":"2026-02-22T19:51:46.461880+0000","last_unstale":"2026-02-22T20:32:49.275302+0000","last_undegraded":"2026-02-22T20:32:49.275302+0000","last_fullsized":"2026-02-22T20:32:49.275302+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:20:16.291609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000706108,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:50.367546+0000","last_change":"2026-02-22T19:51:48.562746+0000","last_active":"2026-02-22T20:32:50.367546+0000","last_peered":"2026-02-22T20:32:50.367546+0000","last_clean":"2026-02-22T20:32:50.367546+0000","last_became_active":"2026-02-22T19:51:46.461871+0000","last_became_peered":"2026-02-22T19:51:46.461871+0000","last_unstale":"2026-02-22T20:32:50.367546+0000","last_undegraded":"2026-02-22T20:32:50.367546+0000","last_fullsized":"2026-02-22T20:32:50.367546+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:23.013120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028334900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:50.368324+0000","last_change":"2026-02-22T19:51:48.562686+0000","last_active":"2026-02-22T20:32:50.368324+0000","last_peered":"2026-02-22T20:32:50.368324+0000","last_clean":"2026-02-22T20:32:50.368324+0000","last_became_active":"2026-02-22T19:51:46.461861+0000","last_became_peered":"2026-02-22T19:51:46.461861+0000","last_unstale":"2026-02-22T20:32:50.368324+0000","last_undegraded":"2026-02-22T20:32:50.368324+0000","last_fullsized":"2026-02-22T20:32:50.368324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:28:58.573469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026117799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.587120+0000","last_change":"2026-02-22T19:51:48.467890+0000","last_active":"2026-02-22T20:32:49.587120+0000","last_peered":"2026-02-22T20:32:49.587120+0000","last_clean":"2026-02-22T20:32:49.587120+0000","last_became_active":"2026-02-22T19:51:46.461352+0000","last_became_peered":"2026-02-22T19:51:46.461352+0000","last_unstale":"2026-02-22T20:32:49.587120+0000","last_undegraded":"2026-02-22T20:32:49.587120+0000","last_fullsized":"2026-02-22T20:32:49.587120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:01:13.797556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082900699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":767,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.583793+0000","last_change":"2026-02-22T19:51:45.322266+0000","last_active":"2026-02-22T20:32:49.583793+0000","last_peered":"2026-02-22T20:32:49.583793+0000","last_clean":"2026-02-22T20:32:49.583793+0000","last_became_active":"2026-02-22T19:51:45.321492+0000","last_became_peered":"2026-02-22T19:51:45.321492+0000","last_unstale":"2026-02-22T20:32:49.583793+0000","last_undegraded":"2026-02-22T20:32:49.583793+0000","last_fullsized":"2026-02-22T20:32:49.583793+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_clean_scrub_stamp":"2026-02-22T19:51:43.450496+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:22:35.569483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706168,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1230312,"kb_used_data":2704,"kb_used_omap":40,"kb_used_meta":1227543,"kb_avail":93141528,"statfs":{"total":96636764160,"available":95376924672,"internally_reserved":0,"allocated":2768896,"data_stored":1458753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41475,"internal_metadata":1257004541},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706168,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761328,"kb_used_data":3288,"kb_used_omap":41,"kb_used_meta":757974,"kb_avail":93610512,"statfs":{"total":96636764160,"available":95857164288,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42838,"internal_metadata":776165546},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738875,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":845040,"kb_used_data":3288,"kb_used_omap":103,"kb_used_meta":841624,"kb_avail":93526800,"statfs":{"total":96636764160,"available":95771443200,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106114,"internal_metadata":861823358},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:33:35.205 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2026-02-22T20:33:35.206 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:33:35.356 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T20:33:35.357 INFO:teuthology.orchestra.run.trial042.stderr:dumped all 2026-02-22T20:33:35.367 INFO:teuthology.orchestra.run.trial042.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2026-02-22T20:33:33.630463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2836680,"kb_used_data":9280,"kb_used_omap":185,"kb_used_meta":2827142,"kb_avail":280278840,"statfs":{"total":289910292480,"available":287005532160,"internally_reserved":0,"allocated":9502720,"data_stored":5557033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":190427,"internal_metadata":2894993445},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001110"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.587232+0000","last_change":"2026-02-22T19:51:48.467655+0000","last_active":"2026-02-22T20:32:49.587232+0000","last_peered":"2026-02-22T20:32:49.587232+0000","last_clean":"2026-02-22T20:32:49.587232+0000","last_became_active":"2026-02-22T19:51:46.461368+0000","last_became_peered":"2026-02-22T19:51:46.461368+0000","last_unstale":"2026-02-22T20:32:49.587232+0000","last_undegraded":"2026-02-22T20:32:49.587232+0000","last_fullsized":"2026-02-22T20:32:49.587232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:54:49.652425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059369800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.589453+0000","last_change":"2026-02-22T19:51:48.467546+0000","last_active":"2026-02-22T20:32:49.589453+0000","last_peered":"2026-02-22T20:32:49.589453+0000","last_clean":"2026-02-22T20:32:49.589453+0000","last_became_active":"2026-02-22T19:51:46.461805+0000","last_became_peered":"2026-02-22T19:51:46.461805+0000","last_unstale":"2026-02-22T20:32:49.589453+0000","last_undegraded":"2026-02-22T20:32:49.589453+0000","last_fullsized":"2026-02-22T20:32:49.589453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:40:08.170741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048713899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.588595+0000","last_change":"2026-02-22T19:51:48.467382+0000","last_active":"2026-02-22T20:32:49.588595+0000","last_peered":"2026-02-22T20:32:49.588595+0000","last_clean":"2026-02-22T20:32:49.588595+0000","last_became_active":"2026-02-22T19:51:46.461823+0000","last_became_peered":"2026-02-22T19:51:46.461823+0000","last_unstale":"2026-02-22T20:32:49.588595+0000","last_undegraded":"2026-02-22T20:32:49.588595+0000","last_fullsized":"2026-02-22T20:32:49.588595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:03:33.088966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000469977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":678,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.586071+0000","last_change":"2026-02-22T19:51:48.468007+0000","last_active":"2026-02-22T20:32:49.586071+0000","last_peered":"2026-02-22T20:32:49.586071+0000","last_clean":"2026-02-22T20:32:49.586071+0000","last_became_active":"2026-02-22T19:51:46.461360+0000","last_became_peered":"2026-02-22T19:51:46.461360+0000","last_unstale":"2026-02-22T20:32:49.586071+0000","last_undegraded":"2026-02-22T20:32:49.586071+0000","last_fullsized":"2026-02-22T20:32:49.586071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:35:55.755801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000927201,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":681,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.275302+0000","last_change":"2026-02-22T19:51:48.465651+0000","last_active":"2026-02-22T20:32:49.275302+0000","last_peered":"2026-02-22T20:32:49.275302+0000","last_clean":"2026-02-22T20:32:49.275302+0000","last_became_active":"2026-02-22T19:51:46.461880+0000","last_became_peered":"2026-02-22T19:51:46.461880+0000","last_unstale":"2026-02-22T20:32:49.275302+0000","last_undegraded":"2026-02-22T20:32:49.275302+0000","last_fullsized":"2026-02-22T20:32:49.275302+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:20:16.291609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000706108,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:50.367546+0000","last_change":"2026-02-22T19:51:48.562746+0000","last_active":"2026-02-22T20:32:50.367546+0000","last_peered":"2026-02-22T20:32:50.367546+0000","last_clean":"2026-02-22T20:32:50.367546+0000","last_became_active":"2026-02-22T19:51:46.461871+0000","last_became_peered":"2026-02-22T19:51:46.461871+0000","last_unstale":"2026-02-22T20:32:50.367546+0000","last_undegraded":"2026-02-22T20:32:50.367546+0000","last_fullsized":"2026-02-22T20:32:50.367546+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:23.013120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028334900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:50.368324+0000","last_change":"2026-02-22T19:51:48.562686+0000","last_active":"2026-02-22T20:32:50.368324+0000","last_peered":"2026-02-22T20:32:50.368324+0000","last_clean":"2026-02-22T20:32:50.368324+0000","last_became_active":"2026-02-22T19:51:46.461861+0000","last_became_peered":"2026-02-22T19:51:46.461861+0000","last_unstale":"2026-02-22T20:32:50.368324+0000","last_undegraded":"2026-02-22T20:32:50.368324+0000","last_fullsized":"2026-02-22T20:32:50.368324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:28:58.573469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026117799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.587120+0000","last_change":"2026-02-22T19:51:48.467890+0000","last_active":"2026-02-22T20:32:49.587120+0000","last_peered":"2026-02-22T20:32:49.587120+0000","last_clean":"2026-02-22T20:32:49.587120+0000","last_became_active":"2026-02-22T19:51:46.461352+0000","last_became_peered":"2026-02-22T19:51:46.461352+0000","last_unstale":"2026-02-22T20:32:49.587120+0000","last_undegraded":"2026-02-22T20:32:49.587120+0000","last_fullsized":"2026-02-22T20:32:49.587120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:01:13.797556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082900699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":767,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.583793+0000","last_change":"2026-02-22T19:51:45.322266+0000","last_active":"2026-02-22T20:32:49.583793+0000","last_peered":"2026-02-22T20:32:49.583793+0000","last_clean":"2026-02-22T20:32:49.583793+0000","last_became_active":"2026-02-22T19:51:45.321492+0000","last_became_peered":"2026-02-22T19:51:45.321492+0000","last_unstale":"2026-02-22T20:32:49.583793+0000","last_undegraded":"2026-02-22T20:32:49.583793+0000","last_fullsized":"2026-02-22T20:32:49.583793+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_clean_scrub_stamp":"2026-02-22T19:51:43.450496+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:22:35.569483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706168,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1230312,"kb_used_data":2704,"kb_used_omap":40,"kb_used_meta":1227543,"kb_avail":93141528,"statfs":{"total":96636764160,"available":95376924672,"internally_reserved":0,"allocated":2768896,"data_stored":1458753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41475,"internal_metadata":1257004541},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706168,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761328,"kb_used_data":3288,"kb_used_omap":41,"kb_used_meta":757974,"kb_avail":93610512,"statfs":{"total":96636764160,"available":95857164288,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42838,"internal_metadata":776165546},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738875,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":845040,"kb_used_data":3288,"kb_used_omap":103,"kb_used_meta":841624,"kb_avail":93526800,"statfs":{"total":96636764160,"available":95771443200,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106114,"internal_metadata":861823358},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:33:35.368 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-22T20:33:35.511 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T20:33:35.511 INFO:teuthology.orchestra.run.trial042.stdout:{"epoch":336,"fsid":"de51f049-31b3-4825-9b8c-01cc5ba58723","created":"2026-02-22T19:51:31.088879+0000","modified":"2026-02-22T20:32:49.268438+0000","last_up_change":"2026-02-22T19:51:41.445931+0000","last_in_change":"2026-02-22T19:51:31.690292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:51:43.260639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:51:45.285569+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"89d5fed7-52dc-4e33-8e1e-4c3ec1b58f68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6802","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6803","nonce":1163499899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6804","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6805","nonce":1163499899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6808","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6809","nonce":1163499899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6806","nonce":1163499899},{"type":"v1","addr":"10.20.193.42:6807","nonce":1163499899}]},"public_addr":"10.20.193.42:6803/1163499899","cluster_addr":"10.20.193.42:6805/1163499899","heartbeat_back_addr":"10.20.193.42:6809/1163499899","heartbeat_front_addr":"10.20.193.42:6807/1163499899","state":["exists","up"]},{"osd":1,"uuid":"3dff05e0-678e-4469-8c56-b41daeb8a87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6810","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6811","nonce":1923107216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6812","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6813","nonce":1923107216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6816","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6817","nonce":1923107216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6814","nonce":1923107216},{"type":"v1","addr":"10.20.193.42:6815","nonce":1923107216}]},"public_addr":"10.20.193.42:6811/1923107216","cluster_addr":"10.20.193.42:6813/1923107216","heartbeat_back_addr":"10.20.193.42:6817/1923107216","heartbeat_front_addr":"10.20.193.42:6815/1923107216","state":["exists","up"]},{"osd":2,"uuid":"eafdcdb0-26b9-4071-80a8-29486388a96e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6818","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6819","nonce":1307521726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6820","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6821","nonce":1307521726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6824","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6825","nonce":1307521726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.42:6822","nonce":1307521726},{"type":"v1","addr":"10.20.193.42:6823","nonce":1307521726}]},"public_addr":"10.20.193.42:6819/1307521726","cluster_addr":"10.20.193.42:6821/1307521726","heartbeat_back_addr":"10.20.193.42:6825/1307521726","heartbeat_front_addr":"10.20.193.42:6823/1307521726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.357125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.513975+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:39.630821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T20:33:36.523 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2026-02-22T20:33:36.523 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:33:36.598 INFO:teuthology.orchestra.run.trial042.stdout:{ 2026-02-22T20:33:36.599 INFO:teuthology.orchestra.run.trial042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T20:33:36.599 INFO:teuthology.orchestra.run.trial042.stdout:} 2026-02-22T20:33:36.608 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2026-02-22T20:33:36.753 INFO:teuthology.orchestra.run.trial042.stderr:instructed osd(s) 0 to deep-scrub 2026-02-22T20:33:36.767 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2026-02-22T20:33:36.767 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:33:36.844 INFO:teuthology.orchestra.run.trial042.stdout:{ 2026-02-22T20:33:36.844 INFO:teuthology.orchestra.run.trial042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:33:36.844 INFO:teuthology.orchestra.run.trial042.stdout:} 2026-02-22T20:33:36.853 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2026-02-22T20:33:36.997 INFO:teuthology.orchestra.run.trial042.stderr:instructed osd(s) 1 to deep-scrub 2026-02-22T20:33:37.009 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2026-02-22T20:33:37.010 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:33:37.092 INFO:teuthology.orchestra.run.trial042.stdout:{ 2026-02-22T20:33:37.093 INFO:teuthology.orchestra.run.trial042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:33:37.093 INFO:teuthology.orchestra.run.trial042.stdout:} 2026-02-22T20:33:37.102 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2026-02-22T20:33:37.256 INFO:teuthology.orchestra.run.trial042.stderr:instructed osd(s) 2 to deep-scrub 2026-02-22T20:33:37.269 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:33:37.429 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T20:33:37.429 INFO:teuthology.orchestra.run.trial042.stderr:dumped all 2026-02-22T20:33:37.442 INFO:teuthology.orchestra.run.trial042.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2026-02-22T20:33:35.630680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2836680,"kb_used_data":9280,"kb_used_omap":185,"kb_used_meta":2827142,"kb_avail":280278840,"statfs":{"total":289910292480,"available":287005532160,"internally_reserved":0,"allocated":9502720,"data_stored":5557033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":190427,"internal_metadata":2894993445},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001153"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.587232+0000","last_change":"2026-02-22T19:51:48.467655+0000","last_active":"2026-02-22T20:32:49.587232+0000","last_peered":"2026-02-22T20:32:49.587232+0000","last_clean":"2026-02-22T20:32:49.587232+0000","last_became_active":"2026-02-22T19:51:46.461368+0000","last_became_peered":"2026-02-22T19:51:46.461368+0000","last_unstale":"2026-02-22T20:32:49.587232+0000","last_undegraded":"2026-02-22T20:32:49.587232+0000","last_fullsized":"2026-02-22T20:32:49.587232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:54:49.652425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059369800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.589453+0000","last_change":"2026-02-22T19:51:48.467546+0000","last_active":"2026-02-22T20:32:49.589453+0000","last_peered":"2026-02-22T20:32:49.589453+0000","last_clean":"2026-02-22T20:32:49.589453+0000","last_became_active":"2026-02-22T19:51:46.461805+0000","last_became_peered":"2026-02-22T19:51:46.461805+0000","last_unstale":"2026-02-22T20:32:49.589453+0000","last_undegraded":"2026-02-22T20:32:49.589453+0000","last_fullsized":"2026-02-22T20:32:49.589453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:40:08.170741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048713899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":677,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.588595+0000","last_change":"2026-02-22T19:51:48.467382+0000","last_active":"2026-02-22T20:32:49.588595+0000","last_peered":"2026-02-22T20:32:49.588595+0000","last_clean":"2026-02-22T20:32:49.588595+0000","last_became_active":"2026-02-22T19:51:46.461823+0000","last_became_peered":"2026-02-22T19:51:46.461823+0000","last_unstale":"2026-02-22T20:32:49.588595+0000","last_undegraded":"2026-02-22T20:32:49.588595+0000","last_fullsized":"2026-02-22T20:32:49.588595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:03:33.088966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000469977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":678,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.586071+0000","last_change":"2026-02-22T19:51:48.468007+0000","last_active":"2026-02-22T20:32:49.586071+0000","last_peered":"2026-02-22T20:32:49.586071+0000","last_clean":"2026-02-22T20:32:49.586071+0000","last_became_active":"2026-02-22T19:51:46.461360+0000","last_became_peered":"2026-02-22T19:51:46.461360+0000","last_unstale":"2026-02-22T20:32:49.586071+0000","last_undegraded":"2026-02-22T20:32:49.586071+0000","last_fullsized":"2026-02-22T20:32:49.586071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:35:55.755801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000927201,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":681,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.275302+0000","last_change":"2026-02-22T19:51:48.465651+0000","last_active":"2026-02-22T20:32:49.275302+0000","last_peered":"2026-02-22T20:32:49.275302+0000","last_clean":"2026-02-22T20:32:49.275302+0000","last_became_active":"2026-02-22T19:51:46.461880+0000","last_became_peered":"2026-02-22T19:51:46.461880+0000","last_unstale":"2026-02-22T20:32:49.275302+0000","last_undegraded":"2026-02-22T20:32:49.275302+0000","last_fullsized":"2026-02-22T20:32:49.275302+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:20:16.291609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000706108,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:50.367546+0000","last_change":"2026-02-22T19:51:48.562746+0000","last_active":"2026-02-22T20:32:50.367546+0000","last_peered":"2026-02-22T20:32:50.367546+0000","last_clean":"2026-02-22T20:32:50.367546+0000","last_became_active":"2026-02-22T19:51:46.461871+0000","last_became_peered":"2026-02-22T19:51:46.461871+0000","last_unstale":"2026-02-22T20:32:50.367546+0000","last_undegraded":"2026-02-22T20:32:50.367546+0000","last_fullsized":"2026-02-22T20:32:50.367546+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:23.013120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028334900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:50.368324+0000","last_change":"2026-02-22T19:51:48.562686+0000","last_active":"2026-02-22T20:32:50.368324+0000","last_peered":"2026-02-22T20:32:50.368324+0000","last_clean":"2026-02-22T20:32:50.368324+0000","last_became_active":"2026-02-22T19:51:46.461861+0000","last_became_peered":"2026-02-22T19:51:46.461861+0000","last_unstale":"2026-02-22T20:32:50.368324+0000","last_undegraded":"2026-02-22T20:32:50.368324+0000","last_fullsized":"2026-02-22T20:32:50.368324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:28:58.573469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026117799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.587120+0000","last_change":"2026-02-22T19:51:48.467890+0000","last_active":"2026-02-22T20:32:49.587120+0000","last_peered":"2026-02-22T20:32:49.587120+0000","last_clean":"2026-02-22T20:32:49.587120+0000","last_became_active":"2026-02-22T19:51:46.461352+0000","last_became_peered":"2026-02-22T19:51:46.461352+0000","last_unstale":"2026-02-22T20:32:49.587120+0000","last_undegraded":"2026-02-22T20:32:49.587120+0000","last_fullsized":"2026-02-22T20:32:49.587120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:45.454643+0000","last_clean_scrub_stamp":"2026-02-22T19:51:45.454643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:01:13.797556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082900699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":767,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:32:49.583793+0000","last_change":"2026-02-22T19:51:45.322266+0000","last_active":"2026-02-22T20:32:49.583793+0000","last_peered":"2026-02-22T20:32:49.583793+0000","last_clean":"2026-02-22T20:32:49.583793+0000","last_became_active":"2026-02-22T19:51:45.321492+0000","last_became_peered":"2026-02-22T19:51:45.321492+0000","last_unstale":"2026-02-22T20:32:49.583793+0000","last_undegraded":"2026-02-22T20:32:49.583793+0000","last_fullsized":"2026-02-22T20:32:49.583793+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:43.450496+0000","last_clean_scrub_stamp":"2026-02-22T19:51:43.450496+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:22:35.569483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706168,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1230312,"kb_used_data":2704,"kb_used_omap":40,"kb_used_meta":1227543,"kb_avail":93141528,"statfs":{"total":96636764160,"available":95376924672,"internally_reserved":0,"allocated":2768896,"data_stored":1458753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41475,"internal_metadata":1257004541},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706168,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761328,"kb_used_data":3288,"kb_used_omap":41,"kb_used_meta":757974,"kb_avail":93610512,"statfs":{"total":96636764160,"available":95857164288,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42838,"internal_metadata":776165546},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738875,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":845040,"kb_used_data":3288,"kb_used_omap":103,"kb_used_meta":841624,"kb_avail":93526800,"statfs":{"total":96636764160,"available":95771443200,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106114,"internal_metadata":861823358},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:33:37.443 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-22T19:51:45.454643+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=45, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=33, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:33:37.444 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-22T19:51:45.454643+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=45, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=33, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:33:37.444 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-22T19:51:45.454643+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=45, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=33, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:33:37.444 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-22T19:51:45.454643+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=45, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=33, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:33:37.444 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-22T19:51:45.454643+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=45, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=33, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:33:37.444 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-22T19:51:45.454643+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=45, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=33, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:33:37.445 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-22T19:51:45.454643+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=45, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=33, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:33:37.445 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-22T19:51:45.454643+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=45, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=33, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:33:37.445 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-22T19:51:43.450496+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=43, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=33, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:33:37.445 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-22T20:33:57.446 DEBUG:teuthology.orchestra.run.trial042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:33:57.605 INFO:teuthology.orchestra.run.trial042.stdout: 2026-02-22T20:33:57.605 INFO:teuthology.orchestra.run.trial042.stderr:dumped all 2026-02-22T20:33:57.616 INFO:teuthology.orchestra.run.trial042.stdout:{"pg_ready":true,"pg_map":{"version":1601,"stamp":"2026-02-22T20:33:55.632781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2836656,"kb_used_data":9280,"kb_used_omap":185,"kb_used_meta":2827142,"kb_avail":280278864,"statfs":{"total":289910292480,"available":287005556736,"internally_reserved":0,"allocated":9502720,"data_stored":5557033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":190427,"internal_metadata":2894993445},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001214"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":686,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:33:38.845874+0000","last_change":"2026-02-22T20:33:38.845895+0000","last_active":"2026-02-22T20:33:38.845874+0000","last_peered":"2026-02-22T20:33:38.845874+0000","last_clean":"2026-02-22T20:33:38.845874+0000","last_became_active":"2026-02-22T19:51:46.461368+0000","last_became_peered":"2026-02-22T19:51:46.461368+0000","last_unstale":"2026-02-22T20:33:38.845874+0000","last_undegraded":"2026-02-22T20:33:38.845874+0000","last_fullsized":"2026-02-22T20:33:38.845874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:33:38.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:33:38.845775+0000","last_clean_scrub_stamp":"2026-02-22T20:33:38.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:17:00.402851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059369800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":686,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:33:40.859573+0000","last_change":"2026-02-22T20:33:40.859589+0000","last_active":"2026-02-22T20:33:40.859573+0000","last_peered":"2026-02-22T20:33:40.859573+0000","last_clean":"2026-02-22T20:33:40.859573+0000","last_became_active":"2026-02-22T19:51:46.461805+0000","last_became_peered":"2026-02-22T19:51:46.461805+0000","last_unstale":"2026-02-22T20:33:40.859573+0000","last_undegraded":"2026-02-22T20:33:40.859573+0000","last_fullsized":"2026-02-22T20:33:40.859573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:33:40.859514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:33:40.859514+0000","last_clean_scrub_stamp":"2026-02-22T20:33:40.859514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:07:42.461066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048713899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":687,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:33:41.829863+0000","last_change":"2026-02-22T20:33:41.829878+0000","last_active":"2026-02-22T20:33:41.829863+0000","last_peered":"2026-02-22T20:33:41.829863+0000","last_clean":"2026-02-22T20:33:41.829863+0000","last_became_active":"2026-02-22T19:51:46.461823+0000","last_became_peered":"2026-02-22T19:51:46.461823+0000","last_unstale":"2026-02-22T20:33:41.829863+0000","last_undegraded":"2026-02-22T20:33:41.829863+0000","last_fullsized":"2026-02-22T20:33:41.829863+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:33:41.829809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:33:41.829809+0000","last_clean_scrub_stamp":"2026-02-22T20:33:41.829809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:08:23.268675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000469977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":687,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:33:39.828777+0000","last_change":"2026-02-22T20:33:39.828795+0000","last_active":"2026-02-22T20:33:39.828777+0000","last_peered":"2026-02-22T20:33:39.828777+0000","last_clean":"2026-02-22T20:33:39.828777+0000","last_became_active":"2026-02-22T19:51:46.461360+0000","last_became_peered":"2026-02-22T19:51:46.461360+0000","last_unstale":"2026-02-22T20:33:39.828777+0000","last_undegraded":"2026-02-22T20:33:39.828777+0000","last_fullsized":"2026-02-22T20:33:39.828777+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:33:39.828705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:33:39.828705+0000","last_clean_scrub_stamp":"2026-02-22T20:33:39.828705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:47:59.487415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000927201,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"336'3","reported_seq":691,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:33:36.959820+0000","last_change":"2026-02-22T20:33:36.959850+0000","last_active":"2026-02-22T20:33:36.959820+0000","last_peered":"2026-02-22T20:33:36.959820+0000","last_clean":"2026-02-22T20:33:36.959820+0000","last_became_active":"2026-02-22T19:51:46.461880+0000","last_became_peered":"2026-02-22T19:51:46.461880+0000","last_unstale":"2026-02-22T20:33:36.959820+0000","last_undegraded":"2026-02-22T20:33:36.959820+0000","last_fullsized":"2026-02-22T20:33:36.959820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'3","last_scrub_stamp":"2026-02-22T20:33:36.959757+0000","last_deep_scrub":"336'3","last_deep_scrub_stamp":"2026-02-22T20:33:36.959757+0000","last_clean_scrub_stamp":"2026-02-22T20:33:36.959757+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:46:08.988591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000706108,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":688,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:33:38.268239+0000","last_change":"2026-02-22T20:33:38.268262+0000","last_active":"2026-02-22T20:33:38.268239+0000","last_peered":"2026-02-22T20:33:38.268239+0000","last_clean":"2026-02-22T20:33:38.268239+0000","last_became_active":"2026-02-22T19:51:46.461871+0000","last_became_peered":"2026-02-22T19:51:46.461871+0000","last_unstale":"2026-02-22T20:33:38.268239+0000","last_undegraded":"2026-02-22T20:33:38.268239+0000","last_fullsized":"2026-02-22T20:33:38.268239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:33:38.268155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:33:38.268155+0000","last_clean_scrub_stamp":"2026-02-22T20:33:38.268155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:25:34.437478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028334900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":688,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:33:37.312528+0000","last_change":"2026-02-22T20:33:37.312547+0000","last_active":"2026-02-22T20:33:37.312528+0000","last_peered":"2026-02-22T20:33:37.312528+0000","last_clean":"2026-02-22T20:33:37.312528+0000","last_became_active":"2026-02-22T19:51:46.461861+0000","last_became_peered":"2026-02-22T19:51:46.461861+0000","last_unstale":"2026-02-22T20:33:37.312528+0000","last_undegraded":"2026-02-22T20:33:37.312528+0000","last_fullsized":"2026-02-22T20:33:37.312528+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:33:37.312457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:33:37.312457+0000","last_clean_scrub_stamp":"2026-02-22T20:33:37.312457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:02:44.022917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00026117799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"336'2","reported_seq":689,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:33:42.789614+0000","last_change":"2026-02-22T20:33:42.789651+0000","last_active":"2026-02-22T20:33:42.789614+0000","last_peered":"2026-02-22T20:33:42.789614+0000","last_clean":"2026-02-22T20:33:42.789614+0000","last_became_active":"2026-02-22T19:51:46.461352+0000","last_became_peered":"2026-02-22T19:51:46.461352+0000","last_unstale":"2026-02-22T20:33:42.789614+0000","last_undegraded":"2026-02-22T20:33:42.789614+0000","last_fullsized":"2026-02-22T20:33:42.789614+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2026-02-22T20:33:42.789518+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2026-02-22T20:33:42.789518+0000","last_clean_scrub_stamp":"2026-02-22T20:33:42.789518+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:46:42.020454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00082900699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"336'74","reported_seq":778,"reported_epoch":336,"state":"active+clean","last_fresh":"2026-02-22T20:33:37.829187+0000","last_change":"2026-02-22T20:33:37.829213+0000","last_active":"2026-02-22T20:33:37.829187+0000","last_peered":"2026-02-22T20:33:37.829187+0000","last_clean":"2026-02-22T20:33:37.829187+0000","last_became_active":"2026-02-22T19:51:45.321492+0000","last_became_peered":"2026-02-22T19:51:45.321492+0000","last_unstale":"2026-02-22T20:33:37.829187+0000","last_undegraded":"2026-02-22T20:33:37.829187+0000","last_fullsized":"2026-02-22T20:33:37.829187+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'74","last_scrub_stamp":"2026-02-22T20:33:37.829096+0000","last_deep_scrub":"336'74","last_deep_scrub_stamp":"2026-02-22T20:33:37.829096+0000","last_clean_scrub_stamp":"2026-02-22T20:33:37.829096+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:19:00.995512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706173,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1230304,"kb_used_data":2704,"kb_used_omap":40,"kb_used_meta":1227543,"kb_avail":93141536,"statfs":{"total":96636764160,"available":95376932864,"internally_reserved":0,"allocated":2768896,"data_stored":1458753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41475,"internal_metadata":1257004541},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706172,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761320,"kb_used_data":3288,"kb_used_omap":41,"kb_used_meta":757974,"kb_avail":93610520,"statfs":{"total":96636764160,"available":95857172480,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42838,"internal_metadata":776165546},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738879,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":845032,"kb_used_data":3288,"kb_used_omap":103,"kb_used_meta":841624,"kb_avail":93526808,"statfs":{"total":96636764160,"available":95771451392,"internally_reserved":0,"allocated":3366912,"data_stored":2049140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106114,"internal_metadata":861823358},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:33:57.618 DEBUG:teuthology.orchestra.run.trial042:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2026-02-22T20:33:57.776 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-22T20:33:57.776 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-22T20:33:57.776 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2026-02-22T20:33:57.776 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:34:00.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2026-02-22T20:34:03.879 INFO:tasks.ceph.cluster1.osd.0:Stopped 2026-02-22T20:34:03.879 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2026-02-22T20:34:03.879 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:34:06.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2026-02-22T20:34:06.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2026-02-22T20:34:09.980 INFO:tasks.ceph.cluster1.osd.1:Stopped 2026-02-22T20:34:09.980 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2026-02-22T20:34:09.980 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:34:11.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2026-02-22T20:34:11.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2026-02-22T20:34:16.082 INFO:tasks.ceph.cluster1.osd.2:Stopped 2026-02-22T20:34:16.083 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-22T20:34:16.083 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2026-02-22T20:34:16.083 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:34:16.124 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2026-02-22T20:34:16.124 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-22T20:34:16.124 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2026-02-22T20:34:16.125 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:34:16.141 INFO:tasks.ceph.cluster1.mon.a:Stopped 2026-02-22T20:34:16.141 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-22T20:34:16.142 DEBUG:teuthology.orchestra.run.trial042:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2026-02-22T20:34:16.198 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@trial042.front.sepia.ceph.com 2026-02-22T20:34:16.198 DEBUG:teuthology.orchestra.run.trial042:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2026-02-22T20:34:16.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2026-02-22T20:34:17.082 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@trial042.front.sepia.ceph.com 2026-02-22T20:34:17.082 DEBUG:teuthology.orchestra.run.trial042:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2026-02-22T20:34:17.849 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@trial042.front.sepia.ceph.com 2026-02-22T20:34:17.850 DEBUG:teuthology.orchestra.run.trial042:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2026-02-22T20:34:18.537 INFO:tasks.ceph:Archiving mon data... 2026-02-22T20:34:18.538 DEBUG:teuthology.misc:Transferring archived files from trial042:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972/data/cluster1.mon.a.tgz 2026-02-22T20:34:18.538 DEBUG:teuthology.orchestra.run.trial042:> mktemp 2026-02-22T20:34:18.542 INFO:teuthology.orchestra.run.trial042.stdout:/tmp/tmp.lo5p6jg9TY 2026-02-22T20:34:18.542 DEBUG:teuthology.orchestra.run.trial042:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.lo5p6jg9TY 2026-02-22T20:34:18.646 DEBUG:teuthology.orchestra.run.trial042:> sudo chmod 0666 /tmp/tmp.lo5p6jg9TY 2026-02-22T20:34:18.705 DEBUG:teuthology.orchestra.remote:trial042:/tmp/tmp.lo5p6jg9TY is 365KB 2026-02-22T20:34:18.762 DEBUG:teuthology.orchestra.run.trial042:> rm -fr /tmp/tmp.lo5p6jg9TY 2026-02-22T20:34:18.766 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-22T20:34:18.767 DEBUG:teuthology.orchestra.run.trial042:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T20:34:18.809 DEBUG:teuthology.orchestra.run.trial066:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T20:34:18.860 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-22T20:34:18.865 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-22T20:34:18.866 INFO:tasks.ceph:Archiving crash dumps... 2026-02-22T20:34:18.868 DEBUG:teuthology.misc:Transferring archived files from trial042:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972/remote/trial042/crash 2026-02-22T20:34:18.870 DEBUG:teuthology.orchestra.run.trial042:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T20:34:18.913 DEBUG:teuthology.misc:Transferring archived files from trial066:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972/remote/trial066/crash 2026-02-22T20:34:18.914 DEBUG:teuthology.orchestra.run.trial066:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T20:34:18.923 INFO:tasks.ceph:Compressing logs... 2026-02-22T20:34:18.924 DEBUG:teuthology.orchestra.run.trial042:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T20:34:18.957 DEBUG:teuthology.orchestra.run.trial066:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T20:34:18.964 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19259.log 2026-02-22T20:34:18.965 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16402.log 2026-02-22T20:34:18.965 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2026-02-22T20:34:18.965 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19259.log.gz 2026-02-22T20:34:18.966 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19175.log 2026-02-22T20:34:18.966 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.16402.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16402.log.gz 2026-02-22T20:34:18.966 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2026-02-22T20:34:18.966 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19175.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19175.log.gz 2026-02-22T20:34:18.966 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2026-02-22T20:34:18.967 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21447.log 2026-02-22T20:34:18.968 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21319.log 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19513.log 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21447.log.gz 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21319.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.12491.log 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21319.log.gz 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21423.log 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19513.log.gz 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/ceph.tmp-client.admin.12491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19260.log 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.12491.log.gz 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21423.log.gz 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19260.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19260.log.gz 2026-02-22T20:34:18.969 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2026-02-22T20:34:18.973 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21189.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-mgr.x.log: /var/log/ceph/cluster2-mgr.x.log: 93.1% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21345.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19070.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21189.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21189.log.gz 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21345.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21345.log.gz 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19095.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19070.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19070.log.gz 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16248.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19095.log.gz 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16155.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.16248.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16248.log.gz 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19416.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19409.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.16155.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16155.log.gz 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21215.log 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19416.log.gz 2026-02-22T20:34:18.974 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19409.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.19409.log.gz 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.19201.log 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19337.log 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21215.log.gz 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19417.log 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19201.log.gz 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19261.log 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19337.log.gz 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19227.log 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr: 87.7% -- replaced with /var/log/ceph/cluster1.log.gz 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19417.log.gz 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19487.log 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19261.log.gz 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19227.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19227.log.gz 2026-02-22T20:34:18.975 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19149.log 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19339.log 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19487.log.gz 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr: 80.8% -- replaced with /var/log/ceph/cluster1-client.admin.19149.log.gz 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19339.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19339.log.gz 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21163.log 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16331.log 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-osd.0.log: /var/log/ceph/cluster1-client.admin.21163.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21163.log.gz 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18818.log 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.16331.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16331.log.gz 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21137.log 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21267.log 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.18818.log: 0.0%/var/log/ceph/cluster1-client.admin.21137.log: -- replaced with /var/log/ceph/cluster1-client.admin.18818.log.gz 2026-02-22T20:34:18.978 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21137.log.gz 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19338.log 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19539.log 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21267.log.gz 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19338.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19338.log.gz 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.19539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21293.log 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19539.log.gz 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16296.log 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21371.log 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21293.log.gz 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.16296.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16296.log.gz 2026-02-22T20:34:18.979 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21397.log 2026-02-22T20:34:18.980 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21241.log 2026-02-22T20:34:18.980 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21371.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21371.log.gz 2026-02-22T20:34:18.980 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16272.log 2026-02-22T20:34:18.981 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.21397.log: /var/log/ceph/cluster1-client.admin.21241.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21397.log.gz 2026-02-22T20:34:18.981 INFO:teuthology.orchestra.run.trial042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21241.log.gz 2026-02-22T20:34:18.981 INFO:teuthology.orchestra.run.trial042.stderr:/var/log/ceph/cluster1-client.admin.16272.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16272.log.gz 2026-02-22T20:34:18.983 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114181.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42602.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114181.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114181.log.gz 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38516.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111543.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120329.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42602.log.gz 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38516.log.gz 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71315.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111543.log.gz 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.120329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82020.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71315.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86647.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71315.log.gz 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82020.log: gzip 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120329.log.gz -5 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.23972.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86647.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82020.log.gz 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35466.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70292.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.23972.log.gz 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98739.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35466.log.gz 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112150.log 2026-02-22T20:34:18.984 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70292.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.98739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124138.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.112150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94142.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98739.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112150.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59250.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123425.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94142.log: /var/log/ceph/cluster2-client.mirror.59250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82860.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59250.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94142.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70199.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123425.log: /var/log/ceph/cluster2-client.mirror.82860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41114.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39165.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123425.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124138.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70199.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128579.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41114.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130212.log 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39165.log.gz 2026-02-22T20:34:18.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.128579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30429.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.130212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43532.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.128579.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82860.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130212.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65394.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30429.log: /var/log/ceph/cluster2-client.mirror.43532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49453.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30429.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43532.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70699.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65394.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114664.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24344.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70699.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49453.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114664.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70699.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56738.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114664.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27506.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41544.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.24344.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56738.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76729.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.27506.log.gz 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33412.log 2026-02-22T20:34:18.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41544.log: /var/log/ceph/cluster2-client.mirror.76729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21426.log 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76729.log.gz 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41544.log.gz 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123996.log 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33412.log.gz 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.21426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129023.log 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19955.log 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123996.log: /var/log/ceph/cluster1-client.mirror.129023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94231.log 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.129023.log.gz 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61572.log 2026-02-22T20:34:18.989 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19955.log.gz 2026-02-22T20:34:18.990 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55303.log 2026-02-22T20:34:18.990 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94231.log.gz 2026-02-22T20:34:18.990 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61572.log: gzip -5 90.7% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.123996.log.gz /var/log/ceph/cluster1-client.mirror.83405.log 2026-02-22T20:34:18.990 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:18.990 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55303.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61572.log.gz 2026-02-22T20:34:18.991 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55303.log.gz 2026-02-22T20:34:18.991 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87072.log 2026-02-22T20:34:18.991 INFO:teuthology.orchestra.run.trial066.stderr:gzip/var/log/ceph/cluster1-client.mirror.83405.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100197.log 2026-02-22T20:34:18.992 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64243.log 2026-02-22T20:34:18.992 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95831.log 2026-02-22T20:34:18.992 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100197.log: /var/log/ceph/cluster2-client.mirror.64243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106724.log 2026-02-22T20:34:18.992 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.100197.log.gz 2026-02-22T20:34:18.992 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64243.log.gz 2026-02-22T20:34:18.992 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.95831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35677.log 2026-02-22T20:34:18.992 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.95831.log.gz 2026-02-22T20:34:18.992 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.106724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83638.log 2026-02-22T20:34:18.993 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70048.log 2026-02-22T20:34:18.993 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35677.log: /var/log/ceph/cluster2-client.mirror.83638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39386.log 2026-02-22T20:34:18.993 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.83405.log.gz 2026-02-22T20:34:18.993 INFO:teuthology.orchestra.run.trial066.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83638.log.gz 2026-02-22T20:34:18.993 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94285.log 2026-02-22T20:34:18.994 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70048.log: /var/log/ceph/cluster2-client.mirror.39386.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106724.log.gz 2026-02-22T20:34:18.994 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52246.log 2026-02-22T20:34:18.994 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39386.log.gz 2026-02-22T20:34:18.994 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70048.log.gz 2026-02-22T20:34:18.994 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107240.log 2026-02-22T20:34:18.994 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.94285.log.gz 2026-02-22T20:34:18.994 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.52246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38051.log 2026-02-22T20:34:18.994 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.107240.log: gzip 88.5% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.52246.log.gz --verbose 2026-02-22T20:34:18.994 INFO:teuthology.orchestra.run.trial066.stderr: -- /var/log/ceph/cluster2-client.mirror.80308.log 2026-02-22T20:34:18.995 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107240.log.gz 2026-02-22T20:34:18.995 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126334.log 2026-02-22T20:34:18.995 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38051.log.gz 2026-02-22T20:34:18.995 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33598.log 2026-02-22T20:34:18.996 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126334.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80308.log.gz 2026-02-22T20:34:18.996 INFO:teuthology.orchestra.run.trial066.stderr: 96.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35677.log.gz 2026-02-22T20:34:18.996 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40986.log 2026-02-22T20:34:18.996 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126334.log.gz 2026-02-22T20:34:18.996 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65080.log 2026-02-22T20:34:18.996 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33598.log: /var/log/ceph/cluster2-client.mirror.40986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35635.log 2026-02-22T20:34:18.996 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40986.log.gz 2026-02-22T20:34:18.996 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33598.log.gz 2026-02-22T20:34:18.997 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130010.log 2026-02-22T20:34:18.997 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35635.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65080.log.gz 2026-02-22T20:34:18.997 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90299.log 2026-02-22T20:34:18.998 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38458.log 2026-02-22T20:34:18.998 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50011.log 2026-02-22T20:34:18.998 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90299.log.gz 2026-02-22T20:34:18.998 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105340.log 2026-02-22T20:34:18.998 INFO:teuthology.orchestra.run.trial066.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130010.log.gz 2026-02-22T20:34:18.998 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38458.log.gz 2026-02-22T20:34:18.999 INFO:teuthology.orchestra.run.trial066.stderr:gzip/var/log/ceph/cluster2-client.mirror.50011.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106217.log 2026-02-22T20:34:18.999 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116554.log 2026-02-22T20:34:18.999 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50011.log.gz 2026-02-22T20:34:18.999 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22484.log 2026-02-22T20:34:18.999 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105340.log.gz 2026-02-22T20:34:19.000 INFO:teuthology.orchestra.run.trial066.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35635.log.gz 2026-02-22T20:34:19.000 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131737.log 2026-02-22T20:34:19.000 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116554.log.gz 2026-02-22T20:34:19.000 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85687.log 2026-02-22T20:34:19.000 INFO:teuthology.orchestra.run.trial066.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106217.log.gz 2026-02-22T20:34:19.000 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.22484.log.gz 2026-02-22T20:34:19.000 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131737.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131737.log.gz 2026-02-22T20:34:19.000 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75495.log 2026-02-22T20:34:19.001 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30208.log 2026-02-22T20:34:19.001 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125816.log 2026-02-22T20:34:19.001 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75495.log: /var/log/ceph/cluster2-client.mirror.30208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87272.log 2026-02-22T20:34:19.001 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30208.log.gz 2026-02-22T20:34:19.001 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.101048.log 2026-02-22T20:34:19.001 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125816.log.gz 2026-02-22T20:34:19.002 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114355.log 2026-02-22T20:34:19.002 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87272.log: /var/log/ceph/cluster1-client.mirror.2.101048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128705.log 2026-02-22T20:34:19.002 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69548.log 2026-02-22T20:34:19.003 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92584.log 2026-02-22T20:34:19.003 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.128705.log.gz 2026-02-22T20:34:19.003 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32819.log 2026-02-22T20:34:19.004 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69548.log.gz 2026-02-22T20:34:19.004 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64092.log 2026-02-22T20:34:19.004 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32819.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92584.log.gz 2026-02-22T20:34:19.004 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32819.log.gz 2026-02-22T20:34:19.004 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88001.log 2026-02-22T20:34:19.004 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21928.log 2026-02-22T20:34:19.005 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64092.log: /var/log/ceph/cluster2-client.mirror.88001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123712.log 2026-02-22T20:34:19.005 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64092.log.gz 2026-02-22T20:34:19.005 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.21928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43253.log 2026-02-22T20:34:19.005 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.21928.log.gz 2026-02-22T20:34:19.005 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124970.log 2026-02-22T20:34:19.006 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19421.log 2026-02-22T20:34:19.006 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43253.log.gz 2026-02-22T20:34:19.006 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81492.log 2026-02-22T20:34:19.006 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19421.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124970.log.gz 2026-02-22T20:34:19.006 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19421.log.gz 2026-02-22T20:34:19.006 INFO:teuthology.orchestra.run.trial066.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123712.log.gz 2026-02-22T20:34:19.007 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66045.log 2026-02-22T20:34:19.007 INFO:teuthology.orchestra.run.trial066.stderr: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75495.log.gz 2026-02-22T20:34:19.007 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89520.log 2026-02-22T20:34:19.007 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81492.log: /var/log/ceph/cluster2-client.mirror.66045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23879.log 2026-02-22T20:34:19.007 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81492.log.gz 2026-02-22T20:34:19.007 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89520.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66045.log.gz 2026-02-22T20:34:19.008 INFO:teuthology.orchestra.run.trial066.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89520.log.gz 2026-02-22T20:34:19.008 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72466.log 2026-02-22T20:34:19.008 INFO:teuthology.orchestra.run.trial066.stderr: 97.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114355.log.gz 2026-02-22T20:34:19.008 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28378.log 2026-02-22T20:34:19.008 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.23879.log.gz 2026-02-22T20:34:19.008 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72466.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122480.log 2026-02-22T20:34:19.009 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72466.log.gz 2026-02-22T20:34:19.009 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90581.log 2026-02-22T20:34:19.009 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28378.log.gz 2026-02-22T20:34:19.009 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101346.log 2026-02-22T20:34:19.009 INFO:teuthology.orchestra.run.trial066.stderr:gzip/var/log/ceph/cluster2-client.mirror.90581.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49546.log 2026-02-22T20:34:19.010 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101346.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90581.log.gz 2026-02-22T20:34:19.010 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131348.log 2026-02-22T20:34:19.010 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.101346.log.gz 2026-02-22T20:34:19.010 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64336.log 2026-02-22T20:34:19.010 INFO:teuthology.orchestra.run.trial066.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122480.log.gz 2026-02-22T20:34:19.010 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131348.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49546.log.gz 2026-02-22T20:34:19.010 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76484.log 2026-02-22T20:34:19.011 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122338.log 2026-02-22T20:34:19.011 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64336.log.gz 2026-02-22T20:34:19.011 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73303.log 2026-02-22T20:34:19.012 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76484.log.gz 2026-02-22T20:34:19.012 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122338.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101427.log 2026-02-22T20:34:19.012 INFO:teuthology.orchestra.run.trial066.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131348.log.gz 2026-02-22T20:34:19.012 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73303.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73303.log.gz 2026-02-22T20:34:19.012 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82645.log 2026-02-22T20:34:19.012 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66789.log 2026-02-22T20:34:19.013 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101427.log: /var/log/ceph/cluster2-client.mirror.82645.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101427.log.gz 2026-02-22T20:34:19.013 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34214.log 2026-02-22T20:34:19.013 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53629.log 2026-02-22T20:34:19.013 INFO:teuthology.orchestra.run.trial066.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.122338.log.gz 2026-02-22T20:34:19.013 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66789.log.gz 2026-02-22T20:34:19.013 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34214.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34214.log.gz 2026-02-22T20:34:19.013 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100367.log 2026-02-22T20:34:19.014 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125456.log 2026-02-22T20:34:19.014 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53629.log.gz 2026-02-22T20:34:19.014 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47775.log 2026-02-22T20:34:19.014 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125456.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.100367.log.gz 2026-02-22T20:34:19.015 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52334.log 2026-02-22T20:34:19.015 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125456.log.gz 2026-02-22T20:34:19.015 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44613.log 2026-02-22T20:34:19.015 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47775.log.gz 2026-02-22T20:34:19.015 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.52334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127054.log 2026-02-22T20:34:19.016 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109429.log 2026-02-22T20:34:19.016 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44613.log.gz 2026-02-22T20:34:19.016 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100895.log 2026-02-22T20:34:19.016 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82645.log.gz 2026-02-22T20:34:19.016 INFO:teuthology.orchestra.run.trial066.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127054.log.gz 2026-02-22T20:34:19.017 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111831.log 2026-02-22T20:34:19.017 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109429.log.gz 2026-02-22T20:34:19.017 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102064.log 2026-02-22T20:34:19.017 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19193.log 2026-02-22T20:34:19.018 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94037.log 2026-02-22T20:34:19.018 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.102064.log.gz 2026-02-22T20:34:19.018 INFO:teuthology.orchestra.run.trial066.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100895.log.gz 2026-02-22T20:34:19.018 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55710.log 2026-02-22T20:34:19.018 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19193.log.gz 2026-02-22T20:34:19.019 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82880.log 2026-02-22T20:34:19.019 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94037.log.gz 2026-02-22T20:34:19.019 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55710.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52334.log.gz 2026-02-22T20:34:19.019 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20130.log 2026-02-22T20:34:19.019 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55710.log.gz 2026-02-22T20:34:19.019 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68746.log 2026-02-22T20:34:19.020 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120500.log 2026-02-22T20:34:19.020 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77044.log 2026-02-22T20:34:19.020 INFO:teuthology.orchestra.run.trial066.stderr: 85.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20130.log.gz 2026-02-22T20:34:19.020 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120500.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68746.log.gz 2026-02-22T20:34:19.020 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85480.log 2026-02-22T20:34:19.020 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120500.log.gz 2026-02-22T20:34:19.021 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20155.log 2026-02-22T20:34:19.021 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105938.log 2026-02-22T20:34:19.021 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85480.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77044.log.gz 2026-02-22T20:34:19.021 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102582.log 2026-02-22T20:34:19.022 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50046.log 2026-02-22T20:34:19.022 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.105938.log.gz 2026-02-22T20:34:19.022 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.102582.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.102582.log.gz 2026-02-22T20:34:19.022 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49732.log 2026-02-22T20:34:19.023 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93663.log 2026-02-22T20:34:19.023 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50046.log: /var/log/ceph/cluster2-client.mirror.49732.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50046.log.gz 2026-02-22T20:34:19.023 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31366.log 2026-02-22T20:34:19.023 INFO:teuthology.orchestra.run.trial066.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49732.log.gz 2026-02-22T20:34:19.023 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93663.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.82880.log.gz 2026-02-22T20:34:19.023 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116434.log 2026-02-22T20:34:19.024 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93663.log.gz 2026-02-22T20:34:19.024 INFO:teuthology.orchestra.run.trial066.stderr: 95.0%/var/log/ceph/cluster2-client.mirror.31366.log: -- replaced with /var/log/ceph/cluster1-client.mirror.20155.log.gz 2026-02-22T20:34:19.024 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122928.log 2026-02-22T20:34:19.024 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31366.log.gz 2026-02-22T20:34:19.024 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114696.log 2026-02-22T20:34:19.024 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87688.log 2026-02-22T20:34:19.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122928.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116434.log.gz 2026-02-22T20:34:19.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.114696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42788.log 2026-02-22T20:34:19.025 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114696.log.gz 2026-02-22T20:34:19.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87688.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122928.log.gz 2026-02-22T20:34:19.025 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64278.log 2026-02-22T20:34:19.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42788.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42788.log.gz 2026-02-22T20:34:19.026 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32575.log 2026-02-22T20:34:19.026 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126424.log 2026-02-22T20:34:19.026 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64278.log.gz 2026-02-22T20:34:19.027 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35543.log 2026-02-22T20:34:19.027 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32575.log.gz 2026-02-22T20:34:19.027 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45020.log 2026-02-22T20:34:19.027 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126424.log: /var/log/ceph/cluster2-client.mirror.35543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103661.log 2026-02-22T20:34:19.027 INFO:teuthology.orchestra.run.trial066.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126424.log.gz 2026-02-22T20:34:19.027 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35543.log.gz 2026-02-22T20:34:19.027 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20240.log 2026-02-22T20:34:19.028 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45020.log.gz 2026-02-22T20:34:19.028 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46194.log 2026-02-22T20:34:19.028 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103661.log.gz 2026-02-22T20:34:19.028 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.20240.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/cluster1-client.admin.20240.log.gz /var/log/ceph/cluster2-client.mirror.85982.log 2026-02-22T20:34:19.028 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.028 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60459.log 2026-02-22T20:34:19.029 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46194.log.gz 2026-02-22T20:34:19.029 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85982.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124777.log 2026-02-22T20:34:19.029 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129948.log 2026-02-22T20:34:19.030 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60459.log: /var/log/ceph/cluster2-client.mirror.124777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124757.log 2026-02-22T20:34:19.030 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60459.log.gz 2026-02-22T20:34:19.030 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96249.log 2026-02-22T20:34:19.030 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.129948.log.gz 2026-02-22T20:34:19.030 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111419.log 2026-02-22T20:34:19.030 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124757.log.gz 2026-02-22T20:34:19.031 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96249.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124777.log.gz 2026-02-22T20:34:19.031 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97750.log 2026-02-22T20:34:19.031 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114629.log 2026-02-22T20:34:19.031 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111419.log.gz 2026-02-22T20:34:19.031 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34991.log 2026-02-22T20:34:19.032 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42695.log 2026-02-22T20:34:19.032 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114629.log.gz 2026-02-22T20:34:19.032 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115499.log 2026-02-22T20:34:19.032 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34991.log.gz 2026-02-22T20:34:19.032 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42695.log: gzip -5 --verbose -- 92.6% /var/log/ceph/cluster2-client.mirror.56214.log 2026-02-22T20:34:19.032 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.97750.log.gz 2026-02-22T20:34:19.032 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42695.log.gz 2026-02-22T20:34:19.033 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38702.log 2026-02-22T20:34:19.033 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115499.log.gz 2026-02-22T20:34:19.033 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60738.log 2026-02-22T20:34:19.033 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56214.log: /var/log/ceph/cluster2-client.mirror.38702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37714.log 2026-02-22T20:34:19.033 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38702.log.gz 2026-02-22T20:34:19.034 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79249.log 2026-02-22T20:34:19.034 INFO:teuthology.orchestra.run.trial066.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.37714.log: -- replaced with /var/log/ceph/cluster2-client.mirror.60738.log.gz 2026-02-22T20:34:19.034 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39758.log 2026-02-22T20:34:19.034 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37714.log.gz 2026-02-22T20:34:19.034 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123325.log 2026-02-22T20:34:19.034 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79249.log.gz 2026-02-22T20:34:19.034 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120778.log 2026-02-22T20:34:19.035 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57948.log 2026-02-22T20:34:19.035 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39758.log.gz 2026-02-22T20:34:19.035 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52699.log 2026-02-22T20:34:19.035 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.120778.log.gz 2026-02-22T20:34:19.035 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125906.log 2026-02-22T20:34:19.035 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57948.log.gz 2026-02-22T20:34:19.036 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73582.log 2026-02-22T20:34:19.036 INFO:teuthology.orchestra.run.trial066.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123325.log.gz 2026-02-22T20:34:19.036 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52699.log.gz 2026-02-22T20:34:19.036 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27169.log 2026-02-22T20:34:19.036 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125906.log.gz 2026-02-22T20:34:19.036 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19575.log 2026-02-22T20:34:19.036 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73582.log.gz 2026-02-22T20:34:19.037 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102853.log 2026-02-22T20:34:19.037 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27169.log.gz 2026-02-22T20:34:19.037 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19575.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56214.log.gz 2026-02-22T20:34:19.037 INFO:teuthology.orchestra.run.trial066.stderr: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.19575.log.gz 2026-02-22T20:34:19.037 INFO:teuthology.orchestra.run.trial066.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46938.log 2026-02-22T20:34:19.037 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97234.log 2026-02-22T20:34:19.037 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51871.log 2026-02-22T20:34:19.038 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46938.log.gz 2026-02-22T20:34:19.038 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129584.log 2026-02-22T20:34:19.038 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51871.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97234.log.gz 2026-02-22T20:34:19.038 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79739.log 2026-02-22T20:34:19.038 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51871.log.gz 2026-02-22T20:34:19.038 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44985.log 2026-02-22T20:34:19.038 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129584.log.gz 2026-02-22T20:34:19.038 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29371.log 2026-02-22T20:34:19.039 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44985.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79739.log.gz 2026-02-22T20:34:19.039 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31610.log 2026-02-22T20:34:19.039 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44985.log.gz 2026-02-22T20:34:19.039 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31738.log 2026-02-22T20:34:19.039 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29371.log.gz 2026-02-22T20:34:19.039 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94801.log 2026-02-22T20:34:19.040 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105172.log 2026-02-22T20:34:19.040 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31610.log.gz 2026-02-22T20:34:19.040 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31738.log.gz 2026-02-22T20:34:19.040 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75473.log 2026-02-22T20:34:19.040 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94801.log.gz 2026-02-22T20:34:19.040 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114741.log 2026-02-22T20:34:19.040 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105172.log.gz 2026-02-22T20:34:19.041 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.75473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47589.log 2026-02-22T20:34:19.041 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.114741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87739.log 2026-02-22T20:34:19.041 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50476.log 2026-02-22T20:34:19.041 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47589.log.gz 2026-02-22T20:34:19.041 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126986.log 2026-02-22T20:34:19.042 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131815.log 2026-02-22T20:34:19.042 INFO:teuthology.orchestra.run.trial066.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114741.log.gz 2026-02-22T20:34:19.042 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50476.log.gz 2026-02-22T20:34:19.042 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125681.log 2026-02-22T20:34:19.042 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126986.log.gz 2026-02-22T20:34:19.042 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77429.log 2026-02-22T20:34:19.042 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131815.log.gz 2026-02-22T20:34:19.043 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125524.log 2026-02-22T20:34:19.043 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125681.log.gz 2026-02-22T20:34:19.043 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52060.log 2026-02-22T20:34:19.043 INFO:teuthology.orchestra.run.trial066.stderr: 89.1%/var/log/ceph/cluster1-client.mirror.125524.log: -- replaced with /var/log/ceph/cluster2-client.mirror.77429.log.gz 2026-02-22T20:34:19.043 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19830.log 2026-02-22T20:34:19.043 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125524.log.gz 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52060.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61207.log 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.19830.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52060.log.gz 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75473.log.gz 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48186.log 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19830.log.gz 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61207.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52792.log 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% 97.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61207.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.102853.log.gz 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51906.log 2026-02-22T20:34:19.044 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48186.log.gz 2026-02-22T20:34:19.045 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56299.log 2026-02-22T20:34:19.045 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52792.log.gz 2026-02-22T20:34:19.045 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82966.log 2026-02-22T20:34:19.045 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51906.log.gz 2026-02-22T20:34:19.045 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115422.log 2026-02-22T20:34:19.045 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67688.log 2026-02-22T20:34:19.046 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32075.log 2026-02-22T20:34:19.046 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115422.log.gz 2026-02-22T20:34:19.046 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.67688.log: --verbose -- /var/log/ceph/cluster2-client.mirror.47907.log 2026-02-22T20:34:19.046 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32075.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113256.log 2026-02-22T20:34:19.046 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67688.log.gz 2026-02-22T20:34:19.047 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32075.log.gz 2026-02-22T20:34:19.047 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107388.log 2026-02-22T20:34:19.047 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47907.log.gz 2026-02-22T20:34:19.047 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113256.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108867.log 2026-02-22T20:34:19.047 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.107388.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113256.log.gz 2026-02-22T20:34:19.047 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65359.log 2026-02-22T20:34:19.047 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108867.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107388.log.gz 2026-02-22T20:34:19.048 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40348.log 2026-02-22T20:34:19.048 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108867.log.gz 2026-02-22T20:34:19.048 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25960.log 2026-02-22T20:34:19.048 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.40348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81616.log 2026-02-22T20:34:19.048 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65359.log.gz 2026-02-22T20:34:19.048 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30955.log 2026-02-22T20:34:19.048 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40348.log.gz 2026-02-22T20:34:19.049 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25960.log.gz 2026-02-22T20:34:19.049 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36545.log 2026-02-22T20:34:19.049 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.82966.log.gz 2026-02-22T20:34:19.049 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81616.log.gz 2026-02-22T20:34:19.049 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108902.log 2026-02-22T20:34:19.049 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30955.log.gz 2026-02-22T20:34:19.049 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.36545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57145.log 2026-02-22T20:34:19.050 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.36545.log.gz 2026-02-22T20:34:19.050 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60180.log 2026-02-22T20:34:19.050 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108902.log.gz 2026-02-22T20:34:19.050 INFO:teuthology.orchestra.run.trial066.stderr: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56299.log.gz 2026-02-22T20:34:19.050 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83680.log 2026-02-22T20:34:19.050 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42137.log 2026-02-22T20:34:19.050 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57145.log.gz 2026-02-22T20:34:19.050 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60180.log.gz 2026-02-22T20:34:19.051 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82493.log 2026-02-22T20:34:19.051 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42137.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87573.log 2026-02-22T20:34:19.051 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83680.log.gz 2026-02-22T20:34:19.051 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42137.log.gz 2026-02-22T20:34:19.051 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111041.log 2026-02-22T20:34:19.051 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65731.log 2026-02-22T20:34:19.052 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87573.log.gz 2026-02-22T20:34:19.052 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.111041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99708.log 2026-02-22T20:34:19.052 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111041.log.gz 2026-02-22T20:34:19.052 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118149.log 2026-02-22T20:34:19.052 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65731.log.gz 2026-02-22T20:34:19.052 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79564.log 2026-02-22T20:34:19.053 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80945.log 2026-02-22T20:34:19.053 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118149.log.gz 2026-02-22T20:34:19.053 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28569.log 2026-02-22T20:34:19.053 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79564.log.gz 2026-02-22T20:34:19.053 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125209.log 2026-02-22T20:34:19.054 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80945.log.gz 2026-02-22T20:34:19.054 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40928.log 2026-02-22T20:34:19.054 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.28569.log.gz 2026-02-22T20:34:19.054 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2026-02-22T20:34:19.054 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125209.log.gz 2026-02-22T20:34:19.054 INFO:teuthology.orchestra.run.trial066.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82493.log.gz 2026-02-22T20:34:19.054 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124948.log 2026-02-22T20:34:19.054 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40928.log.gz 2026-02-22T20:34:19.055 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56385.log 2026-02-22T20:34:19.055 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85863.log 2026-02-22T20:34:19.055 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.56385.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23228.log 2026-02-22T20:34:19.055 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124948.log.gz 2026-02-22T20:34:19.056 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85863.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.56385.log.gz 2026-02-22T20:34:19.056 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124096.log 2026-02-22T20:34:19.056 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85863.log.gz 2026-02-22T20:34:19.056 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118354.log 2026-02-22T20:34:19.056 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.23228.log.gz 2026-02-22T20:34:19.056 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122772.log 2026-02-22T20:34:19.056 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123605.log 2026-02-22T20:34:19.057 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124096.log.gz 2026-02-22T20:34:19.057 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79459.log 2026-02-22T20:34:19.057 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122772.log.gz 2026-02-22T20:34:19.057 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37249.log 2026-02-22T20:34:19.057 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123605.log.gz 2026-02-22T20:34:19.057 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108276.log 2026-02-22T20:34:19.058 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79459.log.gz 2026-02-22T20:34:19.058 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20579.log 2026-02-22T20:34:19.058 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37249.log.gz 2026-02-22T20:34:19.058 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123070.log 2026-02-22T20:34:19.058 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108276.log.gz 2026-02-22T20:34:19.058 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.20579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90869.log 2026-02-22T20:34:19.058 INFO:teuthology.orchestra.run.trial066.stderr: 71.2% -- replaced with /var/log/ceph/cluster2-client.admin.20579.log.gz 2026-02-22T20:34:19.059 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83186.log 2026-02-22T20:34:19.059 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34063.log 2026-02-22T20:34:19.059 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123070.log.gz 2026-02-22T20:34:19.059 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.90869.log.gz 2026-02-22T20:34:19.059 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.83186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54501.log 2026-02-22T20:34:19.060 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124664.log 2026-02-22T20:34:19.060 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34063.log.gz 2026-02-22T20:34:19.060 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131659.log 2026-02-22T20:34:19.060 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54501.log.gz 2026-02-22T20:34:19.060 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34772.log 2026-02-22T20:34:19.061 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27541.log 2026-02-22T20:34:19.061 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131659.log.gz 2026-02-22T20:34:19.061 INFO:teuthology.orchestra.run.trial066.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124664.log.gz 2026-02-22T20:34:19.061 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67161.log 2026-02-22T20:34:19.061 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34772.log.gz 2026-02-22T20:34:19.061 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23449.log 2026-02-22T20:34:19.061 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27541.log.gz 2026-02-22T20:34:19.062 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82516.log 2026-02-22T20:34:19.062 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67161.log.gz 2026-02-22T20:34:19.062 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23449.log: gzip -5 --verbose 88.3% -- /var/log/ceph/cluster1-client.mirror.111764.log -- replaced with /var/log/ceph/cluster2-client.mirror.23449.log.gz 2026-02-22T20:34:19.062 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.063 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82516.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.83186.log.gz 2026-02-22T20:34:19.063 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67405.log 2026-02-22T20:34:19.063 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.111764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109675.log 2026-02-22T20:34:19.063 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111764.log.gz 2026-02-22T20:34:19.063 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67405.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49767.log 2026-02-22T20:34:19.063 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67405.log.gz 2026-02-22T20:34:19.064 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36409.log 2026-02-22T20:34:19.064 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36086.log 2026-02-22T20:34:19.064 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49767.log.gz 2026-02-22T20:34:19.065 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36409.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118220.log 2026-02-22T20:34:19.065 INFO:teuthology.orchestra.run.trial066.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109675.log.gz 2026-02-22T20:34:19.065 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42323.log 2026-02-22T20:34:19.065 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36086.log.gz 2026-02-22T20:34:19.065 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50325.log 2026-02-22T20:34:19.065 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118220.log.gz 2026-02-22T20:34:19.066 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42323.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82516.log.gz 2026-02-22T20:34:19.066 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21963.log 2026-02-22T20:34:19.066 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42323.log.gz 2026-02-22T20:34:19.066 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78619.log 2026-02-22T20:34:19.066 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50325.log.gz 2026-02-22T20:34:19.066 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.21963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47531.log 2026-02-22T20:34:19.066 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.21963.log.gz 2026-02-22T20:34:19.067 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43904.log 2026-02-22T20:34:19.067 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78619.log.gz 2026-02-22T20:34:19.067 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97836.log 2026-02-22T20:34:19.067 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47531.log.gz 2026-02-22T20:34:19.067 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61052.log 2026-02-22T20:34:19.067 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43904.log.gz 2026-02-22T20:34:19.068 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97836.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36409.log.gz 2026-02-22T20:34:19.068 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77114.log 2026-02-22T20:34:19.068 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.97836.log.gz 2026-02-22T20:34:19.068 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86514.log 2026-02-22T20:34:19.068 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61052.log.gz 2026-02-22T20:34:19.068 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19263.log 2026-02-22T20:34:19.068 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77114.log.gz 2026-02-22T20:34:19.069 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111225.log 2026-02-22T20:34:19.069 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124848.log 2026-02-22T20:34:19.069 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19263.log.gz 2026-02-22T20:34:19.069 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35389.log 2026-02-22T20:34:19.069 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111225.log.gz 2026-02-22T20:34:19.069 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27413.log 2026-02-22T20:34:19.070 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41730.log 2026-02-22T20:34:19.070 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35389.log.gz/var/log/ceph/cluster2-client.mirror.27413.log: 2026-02-22T20:34:19.070 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115114.log 2026-02-22T20:34:19.070 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27413.log.gz 2026-02-22T20:34:19.070 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96923.log 2026-02-22T20:34:19.071 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41730.log.gz 2026-02-22T20:34:19.071 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115114.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124848.log.gz 2026-02-22T20:34:19.071 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23414.log 2026-02-22T20:34:19.071 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115114.log.gz 2026-02-22T20:34:19.071 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107455.log 2026-02-22T20:34:19.071 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96923.log.gz 2026-02-22T20:34:19.071 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98659.log 2026-02-22T20:34:19.072 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.23414.log.gz 2026-02-22T20:34:19.072 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.107455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60215.log 2026-02-22T20:34:19.072 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59843.log 2026-02-22T20:34:19.072 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98659.log.gz 2026-02-22T20:34:19.072 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49267.log 2026-02-22T20:34:19.072 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60215.log.gz 2026-02-22T20:34:19.073 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38609.log 2026-02-22T20:34:19.073 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59843.log.gz 2026-02-22T20:34:19.073 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92630.log 2026-02-22T20:34:19.073 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49267.log.gz 2026-02-22T20:34:19.073 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90623.log 2026-02-22T20:34:19.073 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38609.log.gz 2026-02-22T20:34:19.074 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92630.log: gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92630.log.gz 2026-02-22T20:34:19.074 INFO:teuthology.orchestra.run.trial066.stderr: -- /var/log/ceph/cluster2-client.mirror.124118.log 2026-02-22T20:34:19.074 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44334.log 2026-02-22T20:34:19.074 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.90623.log.gz 2026-02-22T20:34:19.075 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124118.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54466.log 2026-02-22T20:34:19.075 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124118.log.gz 2026-02-22T20:34:19.075 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44334.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26111.log 2026-02-22T20:34:19.075 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44334.log.gz 2026-02-22T20:34:19.075 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54466.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96207.log 2026-02-22T20:34:19.076 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54466.log.gz 2026-02-22T20:34:19.076 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130602.log 2026-02-22T20:34:19.076 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26111.log.gz 2026-02-22T20:34:19.076 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96207.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89909.log 2026-02-22T20:34:19.077 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130602.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130602.log.gz 2026-02-22T20:34:19.077 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95691.log 2026-02-22T20:34:19.077 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30766.log 2026-02-22T20:34:19.077 INFO:teuthology.orchestra.run.trial066.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96207.log.gz 2026-02-22T20:34:19.077 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.89909.log.gz 2026-02-22T20:34:19.077 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.95691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126199.log 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95691.log.gz 2026-02-22T20:34:19.077 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.078 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30766.log: gzip -5 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30766.log.gz 2026-02-22T20:34:19.078 INFO:teuthology.orchestra.run.trial066.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.49639.log 2026-02-22T20:34:19.078 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117977.log 86.8% 2026-02-22T20:34:19.078 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126199.log.gz 2026-02-22T20:34:19.078 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51220.log 2026-02-22T20:34:19.079 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49639.log.gz 2026-02-22T20:34:19.079 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128517.log 2026-02-22T20:34:19.079 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117977.log.gz 2026-02-22T20:34:19.079 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99979.log 2026-02-22T20:34:19.079 INFO:teuthology.orchestra.run.trial066.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51220.log.gz 2026-02-22T20:34:19.080 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128517.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34307.log 2026-02-22T20:34:19.080 INFO:teuthology.orchestra.run.trial066.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128517.log.gz 2026-02-22T20:34:19.080 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111260.log 2026-02-22T20:34:19.080 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99979.log.gz 2026-02-22T20:34:19.080 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36838.log 2026-02-22T20:34:19.080 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34307.log.gz 2026-02-22T20:34:19.081 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43346.log 2026-02-22T20:34:19.081 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111260.log.gz 2026-02-22T20:34:19.081 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119274.log 2026-02-22T20:34:19.081 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36838.log.gz 2026-02-22T20:34:19.081 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84384.log 2026-02-22T20:34:19.081 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43346.log.gz 2026-02-22T20:34:19.082 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121583.log 2026-02-22T20:34:19.082 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119274.log.gz 2026-02-22T20:34:19.082 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.84384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57575.log 2026-02-22T20:34:19.082 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72373.log 2026-02-22T20:34:19.083 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129294.log 2026-02-22T20:34:19.083 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121583.log.gz 2026-02-22T20:34:19.083 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57575.log.gz 2026-02-22T20:34:19.083 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80343.log 2026-02-22T20:34:19.083 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72373.log.gz 2026-02-22T20:34:19.083 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127852.log 2026-02-22T20:34:19.083 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56354.log 2026-02-22T20:34:19.083 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129294.log.gz 2026-02-22T20:34:19.084 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80343.log.gz 2026-02-22T20:34:19.084 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127927.log 2026-02-22T20:34:19.084 INFO:teuthology.orchestra.run.trial066.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.127852.log.gz 2026-02-22T20:34:19.084 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.56354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22097.log 2026-02-22T20:34:19.084 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38423.log 2026-02-22T20:34:19.084 INFO:teuthology.orchestra.run.trial066.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.84384.log.gz 2026-02-22T20:34:19.084 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.56354.log.gz 2026-02-22T20:34:19.084 INFO:teuthology.orchestra.run.trial066.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127927.log.gz 2026-02-22T20:34:19.085 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.22097.log: gzip -5 84.0% --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.22097.log.gz -- 2026-02-22T20:34:19.085 INFO:teuthology.orchestra.run.trial066.stderr: /var/log/ceph/cluster2-client.mirror.73547.log 2026-02-22T20:34:19.085 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115618.log 2026-02-22T20:34:19.085 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38423.log.gz 2026-02-22T20:34:19.085 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127960.log 2026-02-22T20:34:19.085 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73547.log.gz 2026-02-22T20:34:19.086 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67068.log 2026-02-22T20:34:19.086 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115618.log.gz 2026-02-22T20:34:19.086 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82774.log 2026-02-22T20:34:19.086 INFO:teuthology.orchestra.run.trial066.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127960.log.gz 2026-02-22T20:34:19.086 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126176.log 2026-02-22T20:34:19.087 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67068.log.gz 2026-02-22T20:34:19.087 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86915.log 2026-02-22T20:34:19.087 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101265.log 2026-02-22T20:34:19.087 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126176.log.gz 2026-02-22T20:34:19.087 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81736.log 2026-02-22T20:34:19.087 INFO:teuthology.orchestra.run.trial066.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86915.log.gz 2026-02-22T20:34:19.088 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58227.log 2026-02-22T20:34:19.088 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101265.log.gz 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.81736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34807.log 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117537.log 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58227.log.gz 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103542.log 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34807.log.gz 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56552.log 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117537.log.gz 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.103542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45392.log 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56552.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.103542.log.gz 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130693.log 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56552.log.gz 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45392.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82774.log.gz 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57203.log 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45392.log.gz 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.130693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120616.log 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130693.log.gz 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60959.log 2026-02-22T20:34:19.091 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57203.log.gz 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32912.log 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120616.log.gz 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.81736.log.gz 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108600.log 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60959.log.gz 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90427.log 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32912.log.gz 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36966.log 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108600.log.gz 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84048.log 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90427.log.gz 2026-02-22T20:34:19.092 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87665.log 2026-02-22T20:34:19.093 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36966.log.gz 2026-02-22T20:34:19.093 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103475.log 2026-02-22T20:34:19.093 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69211.log 2026-02-22T20:34:19.093 INFO:teuthology.orchestra.run.trial066.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87665.log.gz 2026-02-22T20:34:19.093 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112175.log 2026-02-22T20:34:19.093 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103475.log.gz 2026-02-22T20:34:19.094 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55338.log 2026-02-22T20:34:19.094 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69211.log.gz 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.112175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123376.log 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55338.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119324.log 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55338.log.gz 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38237.log 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123376.log.gz 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111508.log 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112175.log.gz 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119324.log.gz 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30615.log 2026-02-22T20:34:19.095 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38237.log.gz 2026-02-22T20:34:19.096 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58692.log 2026-02-22T20:34:19.096 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111508.log.gz 2026-02-22T20:34:19.096 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19289.log 2026-02-22T20:34:19.096 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30615.log.gz 2026-02-22T20:34:19.096 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45671.log 2026-02-22T20:34:19.096 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58692.log.gz 2026-02-22T20:34:19.097 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26204.log 2026-02-22T20:34:19.097 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19289.log.gz 2026-02-22T20:34:19.097 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45671.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23356.log 2026-02-22T20:34:19.097 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45671.log.gz 2026-02-22T20:34:19.097 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127428.log 2026-02-22T20:34:19.097 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26204.log.gz 2026-02-22T20:34:19.098 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31180.log 2026-02-22T20:34:19.098 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.23356.log.gz 2026-02-22T20:34:19.098 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.127428.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113985.log 2026-02-22T20:34:19.098 INFO:teuthology.orchestra.run.trial066.stderr: 83.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127428.log.gz 2026-02-22T20:34:19.098 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117049.log 2026-02-22T20:34:19.098 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31180.log.gz 2026-02-22T20:34:19.099 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83009.log 2026-02-22T20:34:19.099 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113985.log.gz 2026-02-22T20:34:19.099 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20681.log 2026-02-22T20:34:19.099 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117049.log.gz 2026-02-22T20:34:19.099 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.83009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85959.log 2026-02-22T20:34:19.099 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20681.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105584.log 2026-02-22T20:34:19.100 INFO:teuthology.orchestra.run.trial066.stderr: 58.7% -- replaced with /var/log/ceph/cluster2-client.mirror.20681.log.gz 2026-02-22T20:34:19.100 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29743.log 2026-02-22T20:34:19.100 INFO:teuthology.orchestra.run.trial066.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85959.log.gz 2026-02-22T20:34:19.100 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118255.log 2026-02-22T20:34:19.100 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105584.log.gz 2026-02-22T20:34:19.100 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83722.log 2026-02-22T20:34:19.101 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29743.log.gz 2026-02-22T20:34:19.101 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85813.log 2026-02-22T20:34:19.101 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118255.log.gz 2026-02-22T20:34:19.101 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83869.log 2026-02-22T20:34:19.102 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91966.log 2026-02-22T20:34:19.102 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85813.log.gz 2026-02-22T20:34:19.102 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83722.log.gz 2026-02-22T20:34:19.102 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74849.log 2026-02-22T20:34:19.102 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66847.log 2026-02-22T20:34:19.102 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91966.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.83009.log.gz 2026-02-22T20:34:19.103 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.74849.log: -- replaced with /var/log/ceph/cluster2-client.mirror.91966.log.gzgzip 2026-02-22T20:34:19.103 INFO:teuthology.orchestra.run.trial066.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44183.log 2026-02-22T20:34:19.103 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74849.log.gz 2026-02-22T20:34:19.103 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63068.log 2026-02-22T20:34:19.103 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44183.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83869.log.gz 2026-02-22T20:34:19.103 INFO:teuthology.orchestra.run.trial066.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74512.log 2026-02-22T20:34:19.103 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66847.log.gz 2026-02-22T20:34:19.103 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44183.log.gz 2026-02-22T20:34:19.104 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61430.log 2026-02-22T20:34:19.104 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74605.log 2026-02-22T20:34:19.104 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74512.log.gz 2026-02-22T20:34:19.104 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19348.log 2026-02-22T20:34:19.104 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.61430.log: /var/log/ceph/cluster2-client.mirror.74605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52499.log 2026-02-22T20:34:19.105 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74605.log.gz 2026-02-22T20:34:19.105 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.admin.61430.log.gz/var/log/ceph/cluster2-client.admin.19348.log: 2026-02-22T20:34:19.105 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50697.log 2026-02-22T20:34:19.105 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19348.log.gz 2026-02-22T20:34:19.105 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.52499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16381.log 2026-02-22T20:34:19.105 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50697.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.52499.log.gz 2026-02-22T20:34:19.105 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101300.log 2026-02-22T20:34:19.106 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50697.log.gz 2026-02-22T20:34:19.106 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.16381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63627.log 2026-02-22T20:34:19.106 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16381.log.gz 2026-02-22T20:34:19.106 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83899.log 2026-02-22T20:34:19.106 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101300.log.gz 2026-02-22T20:34:19.106 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36780.log 2026-02-22T20:34:19.107 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63627.log.gz 2026-02-22T20:34:19.107 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.83899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127591.log 2026-02-22T20:34:19.107 INFO:teuthology.orchestra.run.trial042.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2026-02-22T20:34:19.107 INFO:teuthology.orchestra.run.trial066.stderr: 86.9%/var/log/ceph/cluster2-client.mirror.36780.log: -- replaced with /var/log/ceph/cluster1-client.mirror.83899.log.gz 2026-02-22T20:34:19.107 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120697.log 2026-02-22T20:34:19.107 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.127591.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60087.log 2026-02-22T20:34:19.107 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36780.log.gz 2026-02-22T20:34:19.108 INFO:teuthology.orchestra.run.trial066.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.127591.log.gz 2026-02-22T20:34:19.108 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100232.log 2026-02-22T20:34:19.108 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60087.log: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109217.log 2026-02-22T20:34:19.108 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.120697.log.gz 2026-02-22T20:34:19.108 INFO:teuthology.orchestra.run.trial066.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.100232.log: gzip -- replaced with /var/log/ceph/cluster2-client.mirror.60087.log.gz 2026-02-22T20:34:19.108 INFO:teuthology.orchestra.run.trial066.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81758.log 2026-02-22T20:34:19.109 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.109217.log: -- replaced with /var/log/ceph/cluster2-client.mirror.100232.log.gz 2026-02-22T20:34:19.109 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48430.log 2026-02-22T20:34:19.109 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109217.log.gz 2026-02-22T20:34:19.109 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75574.log 2026-02-22T20:34:19.109 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43718.log 2026-02-22T20:34:19.110 INFO:teuthology.orchestra.run.trial066.stderr: 88.6%/var/log/ceph/cluster1-client.admin.75574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129078.log 2026-02-22T20:34:19.110 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48430.log.gz 2026-02-22T20:34:19.110 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43718.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.75574.log.gz 2026-02-22T20:34:19.110 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43718.log.gz 2026-02-22T20:34:19.114 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51948.log 2026-02-22T20:34:19.115 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125366.log 2026-02-22T20:34:19.115 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129078.log: 82.8%/var/log/ceph/cluster2-client.mirror.51948.log: -- replaced with /var/log/ceph/cluster1-client.mirror.129078.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122792.log 2026-02-22T20:34:19.115 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.115 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125366.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51948.log.gz 2026-02-22T20:34:19.115 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24995.log 2026-02-22T20:34:19.115 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125366.log.gz 2026-02-22T20:34:19.117 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122792.log: 97.3% 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122792.log.gz 2026-02-22T20:34:19.117 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.81758.log.gz 2026-02-22T20:34:19.117 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19780.log 2026-02-22T20:34:19.117 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24995.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131594.log 2026-02-22T20:34:19.117 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.19780.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.24995.log.gz 2026-02-22T20:34:19.118 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19780.log.gz 2026-02-22T20:34:19.118 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82688.log 2026-02-22T20:34:19.118 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60552.log 2026-02-22T20:34:19.118 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131594.log.gz 2026-02-22T20:34:19.118 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79774.log 2026-02-22T20:34:19.118 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60552.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50790.log 2026-02-22T20:34:19.119 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60552.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93385.log 2026-02-22T20:34:19.119 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.119 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79774.log: /var/log/ceph/cluster2-client.mirror.50790.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79774.log.gz 2026-02-22T20:34:19.119 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50790.log.gz 2026-02-22T20:34:19.119 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51441.log 2026-02-22T20:34:19.120 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93385.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130337.log 2026-02-22T20:34:19.120 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93385.log.gz 2026-02-22T20:34:19.120 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53908.log 2026-02-22T20:34:19.120 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51441.log.gzgzip 2026-02-22T20:34:19.120 INFO:teuthology.orchestra.run.trial066.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35142.log 2026-02-22T20:34:19.121 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130337.log: /var/log/ceph/cluster2-client.mirror.53908.log: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48988.log 2026-02-22T20:34:19.121 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53908.log.gz 2026-02-22T20:34:19.121 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.130337.log.gz 2026-02-22T20:34:19.121 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110580.log 2026-02-22T20:34:19.121 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82688.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35142.log.gz 2026-02-22T20:34:19.121 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.121 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.107693.log 2026-02-22T20:34:19.122 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48988.log: /var/log/ceph/cluster2-client.mirror.110580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74326.log 2026-02-22T20:34:19.122 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.1.107693.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48988.log.gz 2026-02-22T20:34:19.122 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77464.log 2026-02-22T20:34:19.122 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76274.log 2026-02-22T20:34:19.123 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74326.log.gz 2026-02-22T20:34:19.123 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77464.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110580.log.gz 2026-02-22T20:34:19.123 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77464.log.gz 2026-02-22T20:34:19.123 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114936.log 2026-02-22T20:34:19.124 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121351.log 2026-02-22T20:34:19.124 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76274.log: /var/log/ceph/cluster2-client.mirror.114936.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123254.log 2026-02-22T20:34:19.124 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76274.log.gz 2026-02-22T20:34:19.124 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114936.log.gz 2026-02-22T20:34:19.124 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126469.log 2026-02-22T20:34:19.125 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90392.log 2026-02-22T20:34:19.125 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123254.log: /var/log/ceph/cluster1-client.mirror.126469.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126469.log.gz 2026-02-22T20:34:19.125 INFO:teuthology.orchestra.run.trial066.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121351.log.gz 2026-02-22T20:34:19.126 INFO:teuthology.orchestra.run.trial066.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123254.log.gz 2026-02-22T20:34:19.126 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127099.log 2026-02-22T20:34:19.127 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92146.log 2026-02-22T20:34:19.127 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90392.log: /var/log/ceph/cluster1-client.mirror.127099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82903.log 2026-02-22T20:34:19.127 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127099.log.gz 2026-02-22T20:34:19.127 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90392.log.gz 2026-02-22T20:34:19.127 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117165.log 2026-02-22T20:34:19.128 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103609.log 2026-02-22T20:34:19.128 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82903.log: /var/log/ceph/cluster2-client.mirror.117165.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117165.log.gz 2026-02-22T20:34:19.131 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82903.log.gz 2026-02-22T20:34:19.131 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124735.log 2026-02-22T20:34:19.132 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32017.log 2026-02-22T20:34:19.132 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124735.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124735.log.gz 2026-02-22T20:34:19.133 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66940.log 2026-02-22T20:34:19.133 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32017.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32017.log.gz 2026-02-22T20:34:19.134 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99402.log 2026-02-22T20:34:19.134 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66940.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66940.log.gz 2026-02-22T20:34:19.134 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104060.log 2026-02-22T20:34:19.135 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99402.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.99402.log.gz 2026-02-22T20:34:19.135 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44741.log 2026-02-22T20:34:19.137 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104060.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104060.log.gz 2026-02-22T20:34:19.137 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96849.log 2026-02-22T20:34:19.137 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44741.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44741.log.gz 2026-02-22T20:34:19.138 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54222.log 2026-02-22T20:34:19.139 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.96849.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96849.log.gz 2026-02-22T20:34:19.139 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103510.log 2026-02-22T20:34:19.140 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54222.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54222.log.gz 2026-02-22T20:34:19.140 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105087.log 2026-02-22T20:34:19.141 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103510.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103510.log.gz 2026-02-22T20:34:19.141 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29778.log 2026-02-22T20:34:19.141 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105087.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105087.log.gz 2026-02-22T20:34:19.142 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67254.log 2026-02-22T20:34:19.142 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29778.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29778.log.gz 2026-02-22T20:34:19.143 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86308.log 2026-02-22T20:34:19.143 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67254.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67254.log.gz 2026-02-22T20:34:19.144 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61319.log 2026-02-22T20:34:19.144 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.86308.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111831.log.gz 2026-02-22T20:34:19.145 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66661.log 2026-02-22T20:34:19.145 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.61319.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.86308.log.gz 2026-02-22T20:34:19.145 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19730.log 2026-02-22T20:34:19.146 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66661.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66661.log.gz 2026-02-22T20:34:19.146 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83143.log 2026-02-22T20:34:19.147 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.19730.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19730.log.gz 2026-02-22T20:34:19.147 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118798.log 2026-02-22T20:34:19.149 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.83143.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61319.log.gz 2026-02-22T20:34:19.149 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82602.log 2026-02-22T20:34:19.150 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.118798.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.118798.log.gz 2026-02-22T20:34:19.151 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75385.log 2026-02-22T20:34:19.151 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82602.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.83143.log.gz 2026-02-22T20:34:19.151 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68560.log 2026-02-22T20:34:19.152 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.75385.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75385.log.gz 2026-02-22T20:34:19.152 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29185.log 2026-02-22T20:34:19.153 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68560.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68560.log.gz 2026-02-22T20:34:19.154 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35786.log 2026-02-22T20:34:19.154 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29185.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29185.log.gz 2026-02-22T20:34:19.154 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94898.log 2026-02-22T20:34:19.154 INFO:teuthology.orchestra.run.trial066.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82602.log.gz 2026-02-22T20:34:19.155 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68246.log 2026-02-22T20:34:19.156 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94898.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.107693.log.gz 2026-02-22T20:34:19.156 INFO:teuthology.orchestra.run.trial066.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94898.log.gz 2026-02-22T20:34:19.156 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118915.log 2026-02-22T20:34:19.156 INFO:teuthology.orchestra.run.trial042.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2026-02-22T20:34:19.157 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77779.log 2026-02-22T20:34:19.157 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68246.log: /var/log/ceph/cluster2-client.mirror.118915.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68246.log.gz 2026-02-22T20:34:19.157 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118915.log.gz 2026-02-22T20:34:19.157 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71129.log 2026-02-22T20:34:19.158 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77779.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77779.log.gz 2026-02-22T20:34:19.158 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46415.log 2026-02-22T20:34:19.158 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56189.log 2026-02-22T20:34:19.159 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71129.log: /var/log/ceph/cluster2-client.mirror.46415.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71129.log.gz 2026-02-22T20:34:19.160 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46415.log.gz 2026-02-22T20:34:19.160 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97479.log 2026-02-22T20:34:19.160 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52379.log 2026-02-22T20:34:19.160 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.56189.log: /var/log/ceph/cluster2-client.mirror.97479.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56189.log.gz 2026-02-22T20:34:19.160 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97479.log.gz 2026-02-22T20:34:19.160 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21880.log 2026-02-22T20:34:19.160 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83213.log 2026-02-22T20:34:19.172 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52379.log: /var/log/ceph/cluster2-client.mirror.21880.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103609.log.gz 2026-02-22T20:34:19.173 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131151.log 2026-02-22T20:34:19.176 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83213.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83213.log.gz 2026-02-22T20:34:19.177 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51720.log 2026-02-22T20:34:19.190 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44892.log 2026-02-22T20:34:19.191 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51720.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51720.log.gz 2026-02-22T20:34:19.198 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45485.log 2026-02-22T20:34:19.199 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44892.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44892.log.gz 2026-02-22T20:34:19.206 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30150.log 2026-02-22T20:34:19.207 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45485.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45485.log.gz 2026-02-22T20:34:19.219 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108689.log 2026-02-22T20:34:19.219 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30150.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30150.log.gz 2026-02-22T20:34:19.234 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61342.log 2026-02-22T20:34:19.235 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108689.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108689.log.gz 2026-02-22T20:34:19.246 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118636.log 2026-02-22T20:34:19.258 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61342.log: 93.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108241.log 2026-02-22T20:34:19.259 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118636.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118636.log.gz 2026-02-22T20:34:19.266 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61342.log.gz 2026-02-22T20:34:19.266 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67440.log 2026-02-22T20:34:19.267 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108241.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108241.log.gz 2026-02-22T20:34:19.275 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105482.log 2026-02-22T20:34:19.275 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67440.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67440.log.gz 2026-02-22T20:34:19.286 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89337.log 2026-02-22T20:34:19.288 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105482.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105482.log.gz 2026-02-22T20:34:19.288 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99708.log.gz 2026-02-22T20:34:19.289 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116968.log 2026-02-22T20:34:19.302 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110353.log 2026-02-22T20:34:19.303 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116968.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116968.log.gz 2026-02-22T20:34:19.310 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124828.log 2026-02-22T20:34:19.311 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110353.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110353.log.gz 2026-02-22T20:34:19.315 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92146.log.gz 2026-02-22T20:34:19.318 INFO:teuthology.orchestra.run.trial066.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87739.log.gz 2026-02-22T20:34:19.318 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110796.log 2026-02-22T20:34:19.319 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104764.log 2026-02-22T20:34:19.319 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124828.log: /var/log/ceph/cluster2-client.mirror.110796.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124828.log.gz 2026-02-22T20:34:19.322 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.110796.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131181.log 2026-02-22T20:34:19.322 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.322 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88593.log 2026-02-22T20:34:19.322 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.131181.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104764.log.gz 2026-02-22T20:34:19.322 INFO:teuthology.orchestra.run.trial066.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131181.log.gz 2026-02-22T20:34:19.322 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130978.log 2026-02-22T20:34:19.322 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131763.log 2026-02-22T20:34:19.322 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88593.log: /var/log/ceph/cluster1-client.mirror.130978.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88593.log.gz 2026-02-22T20:34:19.323 INFO:teuthology.orchestra.run.trial066.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.130978.log.gz 2026-02-22T20:34:19.325 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78059.log 2026-02-22T20:34:19.325 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84849.log 2026-02-22T20:34:19.325 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131763.log.gz 2026-02-22T20:34:19.325 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27076.log 2026-02-22T20:34:19.325 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84849.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78059.log.gz 2026-02-22T20:34:19.325 INFO:teuthology.orchestra.run.trial066.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84849.log.gz 2026-02-22T20:34:19.325 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58785.log 2026-02-22T20:34:19.326 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66568.log 2026-02-22T20:34:19.326 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27076.log: /var/log/ceph/cluster2-client.mirror.58785.log: 95.8% 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.27076.log.gz 2026-02-22T20:34:19.326 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.87072.log.gz 2026-02-22T20:34:19.326 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58785.log.gz 2026-02-22T20:34:19.326 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72617.log 2026-02-22T20:34:19.341 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114139.log 2026-02-22T20:34:19.341 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66568.log: /var/log/ceph/cluster2-client.mirror.72617.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120743.log 2026-02-22T20:34:19.341 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72617.log.gz 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66568.log.gz 2026-02-22T20:34:19.341 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114139.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58599.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.114139.log.gz 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130799.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120743.log: /var/log/ceph/cluster2-client.mirror.58599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54094.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120743.log.gz 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58599.log.gz 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130799.log: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130799.log.gz 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81403.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126244.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54094.log: /var/log/ceph/cluster2-client.mirror.81403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34156.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81403.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54094.log.gz 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126244.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67874.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126244.log.gz 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114027.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34156.log: /var/log/ceph/cluster2-client.mirror.67874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122176.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67874.log.gz 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.34156.log.gz 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41358.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114027.log.gz 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46043.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122176.log: /var/log/ceph/cluster2-client.mirror.41358.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109464.log 2026-02-22T20:34:19.342 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122176.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41358.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46043.log: 88.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25274.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46043.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63023.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109464.log: /var/log/ceph/cluster2-client.mirror.25274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62714.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.25274.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109464.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.63023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44799.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127799.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62714.log: /var/log/ceph/cluster2-client.mirror.44799.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62714.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44799.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39816.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129254.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127799.log: /var/log/ceph/cluster2-client.mirror.39816.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127799.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39816.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26762.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95075.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129254.log: /var/log/ceph/cluster2-client.mirror.26762.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.26762.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63023.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58355.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.95075.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129254.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42381.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.58355.log: -- replaced with /var/log/ceph/cluster2-client.mirror.95075.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66882.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58355.log.gz 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41021.log 2026-02-22T20:34:19.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42381.log: /var/log/ceph/cluster2-client.mirror.66882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86570.log 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41021.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42381.log.gz 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66882.log.gz 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96026.log 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41021.log.gz 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70234.log 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86570.log.gz 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96182.log 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96026.log: /var/log/ceph/cluster2-client.mirror.70234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33691.log 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr: 88.7%/var/log/ceph/cluster1-client.mirror.96182.log: -- replaced with /var/log/ceph/cluster2-client.mirror.70234.log.gz 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96026.log.gz 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96182.log.gz 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39072.log 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100915.log 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33691.log.gz 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39072.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39909.log 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39072.log.gz 2026-02-22T20:34:19.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.100915.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.100915.log.gz 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25123.log 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28627.log 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39909.log.gz 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25123.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47810.log 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.25123.log.gz 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46752.log 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.28627.log.gz 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26018.log 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47810.log: /var/log/ceph/cluster2-client.mirror.46752.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35932.log 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26018.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46752.log.gz 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47810.log.gz 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26018.log.gz 2026-02-22T20:34:19.346 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62467.log 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19218.log 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35932.log.gz 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62467.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62467.log.gz 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127256.log 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19218.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/cluster2-client.admin.19218.log.gz /var/log/ceph/cluster2-client.mirror.24007.log 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127256.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120403.log 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127256.log.gz 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24007.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.24007.log.gz 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121439.log 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38330.log 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.120403.log.gz 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43590.log 2026-02-22T20:34:19.347 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38330.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38330.log.gz 2026-02-22T20:34:19.348 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106523.log 2026-02-22T20:34:19.348 INFO:teuthology.orchestra.run.trial066.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121439.log.gz 2026-02-22T20:34:19.348 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43590.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43590.log.gz 2026-02-22T20:34:19.348 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65208.log 2026-02-22T20:34:19.349 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83470.log 2026-02-22T20:34:19.349 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106523.log.gz 2026-02-22T20:34:19.349 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65208.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65208.log.gz 2026-02-22T20:34:19.350 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83256.log 2026-02-22T20:34:19.350 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110318.log 2026-02-22T20:34:19.350 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83256.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83470.log.gz 2026-02-22T20:34:19.353 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83256.log.gz 2026-02-22T20:34:19.354 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74291.log 2026-02-22T20:34:19.354 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49209.log 2026-02-22T20:34:19.354 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110318.log.gz 2026-02-22T20:34:19.354 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110555.log 2026-02-22T20:34:19.354 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74291.log.gz 2026-02-22T20:34:19.355 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55489.log 2026-02-22T20:34:19.355 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.110555.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49209.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130182.log 2026-02-22T20:34:19.355 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.355 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110555.log.gz 2026-02-22T20:34:19.356 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53478.log 2026-02-22T20:34:19.356 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46136.log 2026-02-22T20:34:19.356 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55489.log.gz 2026-02-22T20:34:19.356 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53478.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53478.log.gz 2026-02-22T20:34:19.356 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20715.log 2026-02-22T20:34:19.357 INFO:teuthology.orchestra.run.trial066.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130182.log.gz 2026-02-22T20:34:19.357 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20493.log 2026-02-22T20:34:19.357 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46136.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46136.log.gz 2026-02-22T20:34:19.357 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100609.log 2026-02-22T20:34:19.357 INFO:teuthology.orchestra.run.trial066.stderr: 56.3% -- replaced with /var/log/ceph/cluster2-client.mirror.20715.log.gz 2026-02-22T20:34:19.358 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.20493.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.20493.log.gz 2026-02-22T20:34:19.358 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98022.log 2026-02-22T20:34:19.358 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124806.log 2026-02-22T20:34:19.359 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54036.log 2026-02-22T20:34:19.359 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100609.log.gz 2026-02-22T20:34:19.359 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98022.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.98022.log.gz 2026-02-22T20:34:19.359 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124806.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124806.log.gz 2026-02-22T20:34:19.360 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109023.log 2026-02-22T20:34:19.360 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127410.log 2026-02-22T20:34:19.360 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.109023.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54036.log.gz 2026-02-22T20:34:19.361 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128996.log 2026-02-22T20:34:19.361 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109023.log.gz 2026-02-22T20:34:19.361 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.127410.log: 56.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127410.log.gz 2026-02-22T20:34:19.361 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54408.log 2026-02-22T20:34:19.362 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.128996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122012.log 2026-02-22T20:34:19.362 INFO:teuthology.orchestra.run.trial066.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.54408.log: -- replaced with /var/log/ceph/cluster1-client.mirror.128996.log.gz 2026-02-22T20:34:19.362 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54408.log.gz 2026-02-22T20:34:19.362 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103398.log 2026-02-22T20:34:19.363 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40481.log 2026-02-22T20:34:19.363 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103398.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122012.log.gz 2026-02-22T20:34:19.363 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103398.log.gz 2026-02-22T20:34:19.364 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53292.log 2026-02-22T20:34:19.364 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38551.log 2026-02-22T20:34:19.364 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53292.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53292.log.gz 2026-02-22T20:34:19.365 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100644.log 2026-02-22T20:34:19.365 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38551.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38551.log.gz 2026-02-22T20:34:19.375 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36317.log 2026-02-22T20:34:19.375 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55617.log 2026-02-22T20:34:19.375 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100644.log.gz 2026-02-22T20:34:19.375 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36317.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109541.log 2026-02-22T20:34:19.376 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55617.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36317.log.gz 2026-02-22T20:34:19.376 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55617.log.gz 2026-02-22T20:34:19.377 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106984.log 2026-02-22T20:34:19.377 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39537.log 2026-02-22T20:34:19.377 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109541.log.gz 2026-02-22T20:34:19.378 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71222.log 2026-02-22T20:34:19.378 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39537.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106984.log.gz 2026-02-22T20:34:19.378 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39537.log.gz 2026-02-22T20:34:19.378 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85687.log.gz 2026-02-22T20:34:19.379 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129342.log 2026-02-22T20:34:19.379 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128455.log 2026-02-22T20:34:19.379 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71222.log: /var/log/ceph/cluster2-client.mirror.129342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117403.log 2026-02-22T20:34:19.379 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128455.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71222.log.gz 2026-02-22T20:34:19.380 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92214.log 2026-02-22T20:34:19.380 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117403.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128455.log.gz 2026-02-22T20:34:19.381 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47403.log 2026-02-22T20:34:19.381 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92214.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117403.log.gz 2026-02-22T20:34:19.381 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92214.log.gz 2026-02-22T20:34:19.381 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126829.log 2026-02-22T20:34:19.382 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30708.log 2026-02-22T20:34:19.383 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47403.log: 88.3%/var/log/ceph/cluster1-client.mirror.126829.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47403.log.gz 2026-02-22T20:34:19.383 INFO:teuthology.orchestra.run.trial066.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126829.log.gz 2026-02-22T20:34:19.383 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121534.log 2026-02-22T20:34:19.383 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59622.log 2026-02-22T20:34:19.384 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30708.log: /var/log/ceph/cluster2-client.mirror.121534.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30708.log.gz 2026-02-22T20:34:19.384 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121534.log.gz 2026-02-22T20:34:19.384 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130420.log 2026-02-22T20:34:19.384 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130947.log 2026-02-22T20:34:19.384 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59622.log: /var/log/ceph/cluster1-client.mirror.130420.log: 88.8% 84.8% -- replaced with /var/log/ceph/cluster1-client.mirror.130420.log.gz 2026-02-22T20:34:19.384 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.59622.log.gz 2026-02-22T20:34:19.384 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44520.log 2026-02-22T20:34:19.386 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130947.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130947.log.gz 2026-02-22T20:34:19.386 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98784.log 2026-02-22T20:34:19.387 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44520.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44520.log.gz 2026-02-22T20:34:19.387 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29592.log 2026-02-22T20:34:19.388 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.98784.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98784.log.gz 2026-02-22T20:34:19.389 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55210.log 2026-02-22T20:34:19.389 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29592.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29592.log.gz 2026-02-22T20:34:19.390 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68374.log 2026-02-22T20:34:19.390 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55210.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55210.log.gz 2026-02-22T20:34:19.391 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26576.log 2026-02-22T20:34:19.391 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68374.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68374.log.gz 2026-02-22T20:34:19.392 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97770.log 2026-02-22T20:34:19.392 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26576.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.26576.log.gz 2026-02-22T20:34:19.393 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90469.log 2026-02-22T20:34:19.394 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.97770.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97770.log.gz 2026-02-22T20:34:19.394 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129404.log 2026-02-22T20:34:19.395 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90469.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.90469.log.gz 2026-02-22T20:34:19.395 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123925.log 2026-02-22T20:34:19.396 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129404.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.129404.log.gz 2026-02-22T20:34:19.396 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128746.log 2026-02-22T20:34:19.398 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123925.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123925.log.gz 2026-02-22T20:34:19.398 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75353.log 2026-02-22T20:34:19.400 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128746.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128746.log.gz 2026-02-22T20:34:19.400 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61758.log 2026-02-22T20:34:19.401 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75353.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75353.log.gz 2026-02-22T20:34:19.401 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20442.log 2026-02-22T20:34:19.402 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61758.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61758.log.gz 2026-02-22T20:34:19.402 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47345.log 2026-02-22T20:34:19.403 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20442.log: 64.7% -- replaced with /var/log/ceph/cluster1-client.mirror.20442.log.gz 2026-02-22T20:34:19.403 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129234.log 2026-02-22T20:34:19.404 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47345.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47345.log.gz 2026-02-22T20:34:19.404 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34249.log 2026-02-22T20:34:19.405 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129234.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129234.log.gz 2026-02-22T20:34:19.405 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73768.log 2026-02-22T20:34:19.406 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34249.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34249.log.gz 2026-02-22T20:34:19.406 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122296.log 2026-02-22T20:34:19.407 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73768.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73768.log.gz 2026-02-22T20:34:19.407 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114587.log 2026-02-22T20:34:19.408 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122296.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122296.log.gz 2026-02-22T20:34:19.408 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122105.log 2026-02-22T20:34:19.409 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114587.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114587.log.gz 2026-02-22T20:34:19.409 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122083.log 2026-02-22T20:34:19.410 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122105.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122105.log.gz 2026-02-22T20:34:19.410 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45857.log 2026-02-22T20:34:19.410 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58134.log 2026-02-22T20:34:19.411 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122083.log: /var/log/ceph/cluster2-client.mirror.45857.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45857.log.gz 2026-02-22T20:34:19.411 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122083.log.gz 2026-02-22T20:34:19.412 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128808.log 2026-02-22T20:34:19.412 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58134.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127447.log 2026-02-22T20:34:19.412 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58134.log.gz/var/log/ceph/cluster2-client.mirror.128808.log: 2026-02-22T20:34:19.413 INFO:teuthology.orchestra.run.trial066.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.128808.log.gz 2026-02-22T20:34:19.413 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36163.log 2026-02-22T20:34:19.413 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.127447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124451.log 2026-02-22T20:34:19.413 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.127447.log.gz 2026-02-22T20:34:19.413 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36163.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45543.log 2026-02-22T20:34:19.413 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36163.log.gz 2026-02-22T20:34:19.413 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93081.log 2026-02-22T20:34:19.414 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45543.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124451.log.gz 2026-02-22T20:34:19.414 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45543.log.gz 2026-02-22T20:34:19.414 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122267.log 2026-02-22T20:34:19.414 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.93081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66324.log 2026-02-22T20:34:19.415 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122267.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.93081.log.gz 2026-02-22T20:34:19.416 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116349.log 2026-02-22T20:34:19.416 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66324.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122267.log.gz 2026-02-22T20:34:19.416 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66324.log.gz 2026-02-22T20:34:19.416 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25739.log 2026-02-22T20:34:19.417 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63003.log 2026-02-22T20:34:19.417 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116349.log: /var/log/ceph/cluster2-client.mirror.25739.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116349.log.gz 2026-02-22T20:34:19.417 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25739.log.gz 2026-02-22T20:34:19.417 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86491.log 2026-02-22T20:34:19.418 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74012.log 2026-02-22T20:34:19.418 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63003.log: /var/log/ceph/cluster2-client.mirror.86491.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86491.log.gz 2026-02-22T20:34:19.418 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102029.log 2026-02-22T20:34:19.419 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74012.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74012.log.gz 2026-02-22T20:34:19.419 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44148.log 2026-02-22T20:34:19.420 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102029.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102029.log.gz 2026-02-22T20:34:19.420 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53664.log 2026-02-22T20:34:19.421 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44148.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44148.log.gz 2026-02-22T20:34:19.421 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71594.log 2026-02-22T20:34:19.421 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63003.log.gz 2026-02-22T20:34:19.422 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60924.log 2026-02-22T20:34:19.422 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53664.log.gz 2026-02-22T20:34:19.422 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25402.log 2026-02-22T20:34:19.422 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60924.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71594.log.gz 2026-02-22T20:34:19.423 INFO:teuthology.orchestra.run.trial066.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46659.log 2026-02-22T20:34:19.423 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60924.log.gz 2026-02-22T20:34:19.423 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75994.log 2026-02-22T20:34:19.424 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46659.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.25402.log.gz 2026-02-22T20:34:19.424 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46659.log.gz 2026-02-22T20:34:19.424 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126806.log 2026-02-22T20:34:19.424 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29964.log 2026-02-22T20:34:19.424 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75994.log: /var/log/ceph/cluster1-client.mirror.126806.log: 87.0% 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75994.log.gz 2026-02-22T20:34:19.424 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126806.log.gz 2026-02-22T20:34:19.425 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64464.log 2026-02-22T20:34:19.425 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77849.log 2026-02-22T20:34:19.425 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29964.log: /var/log/ceph/cluster2-client.mirror.64464.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29964.log.gz 2026-02-22T20:34:19.425 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64464.log.gz 2026-02-22T20:34:19.426 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.112448.log 2026-02-22T20:34:19.426 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77849.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77849.log.gz 2026-02-22T20:34:19.427 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32633.log 2026-02-22T20:34:19.427 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93501.log 2026-02-22T20:34:19.427 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.1.112448.log: /var/log/ceph/cluster2-client.mirror.32633.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32633.log.gz 2026-02-22T20:34:19.435 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73117.log 2026-02-22T20:34:19.435 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93501.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.93501.log.gz 2026-02-22T20:34:19.436 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28436.log 2026-02-22T20:34:19.436 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73117.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73117.log.gz 2026-02-22T20:34:19.436 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36044.log 2026-02-22T20:34:19.437 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28436.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28436.log.gz 2026-02-22T20:34:19.437 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54187.log 2026-02-22T20:34:19.438 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36044.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36044.log.gz 2026-02-22T20:34:19.438 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127469.log 2026-02-22T20:34:19.439 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54187.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54187.log.gz 2026-02-22T20:34:19.439 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96871.log 2026-02-22T20:34:19.439 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.127469.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127469.log.gz 2026-02-22T20:34:19.440 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77639.log 2026-02-22T20:34:19.445 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96871.log: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.101048.log.gz 2026-02-22T20:34:19.445 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74047.log 2026-02-22T20:34:19.446 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77639.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77639.log.gz 2026-02-22T20:34:19.446 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2026-02-22T20:34:19.447 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74047.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.112448.log.gz 2026-02-22T20:34:19.447 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74047.log.gz 2026-02-22T20:34:19.447 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59529.log 2026-02-22T20:34:19.448 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-mon.a.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86514.log.gz 2026-02-22T20:34:19.448 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109977.log 2026-02-22T20:34:19.448 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131919.log 2026-02-22T20:34:19.448 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59529.log.gz 2026-02-22T20:34:19.448 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94918.log 2026-02-22T20:34:19.449 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37528.log 2026-02-22T20:34:19.449 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131919.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131919.log.gz 2026-02-22T20:34:19.449 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.94918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91788.log 2026-02-22T20:34:19.449 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37528.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.109977.log.gz 2026-02-22T20:34:19.450 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37528.log.gz 2026-02-22T20:34:19.450 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77254.log 2026-02-22T20:34:19.450 INFO:teuthology.orchestra.run.trial066.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94918.log.gz 2026-02-22T20:34:19.451 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41207.log 2026-02-22T20:34:19.451 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91788.log.gz 2026-02-22T20:34:19.452 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126491.log 2026-02-22T20:34:19.452 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41207.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77254.log.gz 2026-02-22T20:34:19.452 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41207.log.gz 2026-02-22T20:34:19.452 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44648.log 2026-02-22T20:34:19.452 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56645.log 2026-02-22T20:34:19.452 INFO:teuthology.orchestra.run.trial066.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126491.log.gz 2026-02-22T20:34:19.453 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44648.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44648.log.gz 2026-02-22T20:34:19.453 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71501.log 2026-02-22T20:34:19.453 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121040.log 2026-02-22T20:34:19.453 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56645.log.gz 2026-02-22T20:34:19.454 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71501.log: 89.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102207.log 2026-02-22T20:34:19.454 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.71501.log.gz 2026-02-22T20:34:19.455 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121040.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92549.log 2026-02-22T20:34:19.455 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121040.log.gz 2026-02-22T20:34:19.455 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102207.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83428.log 2026-02-22T20:34:19.456 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.92549.log: -- replaced with /var/log/ceph/cluster2-client.mirror.102207.log.gz 2026-02-22T20:34:19.456 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92549.log.gz 2026-02-22T20:34:19.456 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58076.log 2026-02-22T20:34:19.456 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83764.log 2026-02-22T20:34:19.457 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83428.log: /var/log/ceph/cluster2-client.mirror.58076.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58076.log.gz 2026-02-22T20:34:19.457 INFO:teuthology.orchestra.run.trial066.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83428.log.gz 2026-02-22T20:34:19.457 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85278.log 2026-02-22T20:34:19.457 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129524.log 2026-02-22T20:34:19.457 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85278.log: 84.7% 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83764.log.gz 2026-02-22T20:34:19.458 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.85278.log.gz 2026-02-22T20:34:19.458 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93843.log 2026-02-22T20:34:19.458 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67533.log 2026-02-22T20:34:19.458 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129524.log.gz/var/log/ceph/cluster2-client.mirror.93843.log: 2026-02-22T20:34:19.459 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116469.log 2026-02-22T20:34:19.459 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67533.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67533.log.gz 2026-02-22T20:34:19.459 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73268.log 2026-02-22T20:34:19.460 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116469.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116469.log.gz 2026-02-22T20:34:19.460 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92281.log 2026-02-22T20:34:19.461 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73268.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73268.log.gz 2026-02-22T20:34:19.461 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121725.log 2026-02-22T20:34:19.462 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.92281.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92281.log.gz 2026-02-22T20:34:19.462 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34714.log 2026-02-22T20:34:19.463 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121725.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121725.log.gz 2026-02-22T20:34:19.463 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117761.log 2026-02-22T20:34:19.464 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34714.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34714.log.gz 2026-02-22T20:34:19.464 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37621.log 2026-02-22T20:34:19.464 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117761.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117761.log.gz 2026-02-22T20:34:19.465 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96511.log 2026-02-22T20:34:19.465 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37621.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37621.log.gz 2026-02-22T20:34:19.466 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40436.log 2026-02-22T20:34:19.467 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96511.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96511.log.gz 2026-02-22T20:34:19.467 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23263.log 2026-02-22T20:34:19.470 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.40436.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40436.log.gz 2026-02-22T20:34:19.471 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97020.log 2026-02-22T20:34:19.471 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23263.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.23263.log.gz 2026-02-22T20:34:19.471 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96829.log 2026-02-22T20:34:19.472 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97020.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97020.log.gz 2026-02-22T20:34:19.472 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122460.log 2026-02-22T20:34:19.473 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96829.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96829.log.gz 2026-02-22T20:34:19.474 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61461.log 2026-02-22T20:34:19.474 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122460.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122460.log.gz 2026-02-22T20:34:19.475 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39293.log 2026-02-22T20:34:19.475 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.61461.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.61461.log.gz 2026-02-22T20:34:19.476 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46473.log 2026-02-22T20:34:19.476 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107455.log.gz 2026-02-22T20:34:19.476 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39293.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39293.log.gz 2026-02-22T20:34:19.476 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36745.log 2026-02-22T20:34:19.477 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31645.log 2026-02-22T20:34:19.477 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36745.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46473.log.gz 2026-02-22T20:34:19.477 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36745.log.gz 2026-02-22T20:34:19.477 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115248.log 2026-02-22T20:34:19.478 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47124.log 2026-02-22T20:34:19.478 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31645.log.gz 2026-02-22T20:34:19.478 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84496.log 2026-02-22T20:34:19.479 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47124.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47124.log.gz 2026-02-22T20:34:19.479 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39351.log 2026-02-22T20:34:19.494 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124238.log 2026-02-22T20:34:19.495 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39351.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39351.log.gz 2026-02-22T20:34:19.502 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78689.log 2026-02-22T20:34:19.503 INFO:teuthology.orchestra.run.trial042.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2026-02-22T20:34:19.503 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124238.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124238.log.gz 2026-02-22T20:34:19.510 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20562.log 2026-02-22T20:34:19.511 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78689.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78689.log.gz 2026-02-22T20:34:19.518 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29406.log 2026-02-22T20:34:19.530 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.20562.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.admin.20562.log.gz 2026-02-22T20:34:19.534 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131151.log.gz 2026-02-22T20:34:19.534 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104704.log 2026-02-22T20:34:19.535 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122899.log 2026-02-22T20:34:19.535 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29406.log.gz 2026-02-22T20:34:19.548 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104704.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93843.log.gz 2026-02-22T20:34:19.550 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40835.log 2026-02-22T20:34:19.552 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122899.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122899.log.gz 2026-02-22T20:34:19.553 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124309.log 2026-02-22T20:34:19.553 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40835.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40835.log.gz 2026-02-22T20:34:19.553 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126761.log 2026-02-22T20:34:19.554 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124309.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124309.log.gz 2026-02-22T20:34:19.554 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101556.log 2026-02-22T20:34:19.554 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126761.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126761.log.gz 2026-02-22T20:34:19.555 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60494.log 2026-02-22T20:34:19.555 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101556.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101556.log.gz 2026-02-22T20:34:19.556 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129503.log 2026-02-22T20:34:19.556 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60494.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60494.log.gz 2026-02-22T20:34:19.557 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88304.log 2026-02-22T20:34:19.557 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129503.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129503.log.gz 2026-02-22T20:34:19.557 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66382.log 2026-02-22T20:34:19.558 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88304.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88304.log.gz 2026-02-22T20:34:19.558 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119458.log 2026-02-22T20:34:19.558 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66382.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66382.log.gz 2026-02-22T20:34:19.559 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81716.log 2026-02-22T20:34:19.566 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31889.log 2026-02-22T20:34:19.569 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81716.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96249.log.gz 2026-02-22T20:34:19.570 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81716.log.gz 2026-02-22T20:34:19.570 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114104.log 2026-02-22T20:34:19.571 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31889.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31889.log.gz 2026-02-22T20:34:19.572 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127684.log 2026-02-22T20:34:19.572 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114104.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114104.log.gz 2026-02-22T20:34:19.583 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121818.log 2026-02-22T20:34:19.584 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127684.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127684.log.gz 2026-02-22T20:34:19.584 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102331.log 2026-02-22T20:34:19.584 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121818.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121818.log.gz 2026-02-22T20:34:19.584 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130235.log 2026-02-22T20:34:19.585 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102331.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.102331.log.gz 2026-02-22T20:34:19.585 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80424.log 2026-02-22T20:34:19.585 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.130235.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.130235.log.gz 2026-02-22T20:34:19.586 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39479.log 2026-02-22T20:34:19.589 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.80424.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.80424.log.gz 2026-02-22T20:34:19.589 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84332.log 2026-02-22T20:34:19.590 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39479.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39479.log.gz 2026-02-22T20:34:19.590 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113345.log 2026-02-22T20:34:19.591 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84332.log: 84.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84332.log.gz 2026-02-22T20:34:19.591 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126559.log 2026-02-22T20:34:19.592 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113345.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113345.log.gz 2026-02-22T20:34:19.592 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56277.log 2026-02-22T20:34:19.593 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126559.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126559.log.gz 2026-02-22T20:34:19.593 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84655.log 2026-02-22T20:34:19.596 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.56277.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56277.log.gz 2026-02-22T20:34:19.596 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68002.log 2026-02-22T20:34:19.597 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84655.log: 84.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84655.log.gz 2026-02-22T20:34:19.597 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92969.log 2026-02-22T20:34:19.598 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68002.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68002.log.gz 2026-02-22T20:34:19.598 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82559.log 2026-02-22T20:34:19.598 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92969.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92969.log.gz 2026-02-22T20:34:19.599 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45078.log 2026-02-22T20:34:19.602 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82559.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82559.log.gz 2026-02-22T20:34:19.603 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54745.log 2026-02-22T20:34:19.604 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45078.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45078.log.gz 2026-02-22T20:34:19.614 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87688.log.gz 2026-02-22T20:34:19.614 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125321.log 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41393.log 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125321.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54745.log.gz 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.125321.log.gz 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108635.log 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59343.log 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr: 88.6%/var/log/ceph/cluster2-client.mirror.108635.log: -- replaced with /var/log/ceph/cluster2-client.mirror.41393.log.gz 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118354.log.gz 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108635.log.gz 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40742.log 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25088.log 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.40742.log: -- replaced with /var/log/ceph/cluster2-client.mirror.59343.log.gz 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42753.log 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr: 88.4%/var/log/ceph/cluster2-client.mirror.25088.log: -- replaced with /var/log/ceph/cluster2-client.mirror.40742.log.gz 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123954.log 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99321.log 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.42753.log: -- replaced with /var/log/ceph/cluster2-client.mirror.25088.log.gz 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr: 88.7%/var/log/ceph/cluster2-client.mirror.123954.log: -- replaced with /var/log/ceph/cluster2-client.mirror.42753.log.gz 2026-02-22T20:34:19.630 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125884.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96871.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107113.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123954.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99321.log: /var/log/ceph/cluster1-client.mirror.125884.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123496.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125884.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99321.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.107113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33470.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123496.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107113.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130157.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51162.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123496.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.130157.log: -- replaced with /var/log/ceph/cluster2-client.mirror.33470.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69676.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.130157.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77989.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51162.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24379.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77989.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69676.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62791.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 89.1%/var/log/ceph/cluster2-client.mirror.24379.log: -- replaced with /var/log/ceph/cluster2-client.mirror.77989.log.gzgzip 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49674.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.24379.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43439.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30336.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62791.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49674.log: /var/log/ceph/cluster2-client.mirror.43439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122225.log 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.30336.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49674.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43439.log.gz 2026-02-22T20:34:19.631 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30336.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100870.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97101.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105619.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.100870.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.122225.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81581.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97101.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.100870.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105619.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97101.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105619.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54931.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82665.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54931.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81581.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104485.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54931.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51813.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84779.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24530.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104485.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51813.log.gz/var/log/ceph/cluster2-client.mirror.84779.log: 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84779.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42044.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63685.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24530.log: /var/log/ceph/cluster2-client.mirror.42044.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81861.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.24530.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42044.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63685.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63685.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97970.log 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81861.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.82665.log.gz 2026-02-22T20:34:19.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122125.log 2026-02-22T20:34:19.633 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81861.log.gz 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124380.log 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82536.log 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126109.log 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82536.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124380.log.gz 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122125.log.gz 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121796.log 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84608.log 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126109.log.gz 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67909.log 2026-02-22T20:34:19.635 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84608.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121796.log.gz 2026-02-22T20:34:19.636 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48151.log 2026-02-22T20:34:19.636 INFO:teuthology.orchestra.run.trial066.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82536.log.gz 2026-02-22T20:34:19.636 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67909.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67909.log.gz 2026-02-22T20:34:19.646 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84048.log.gz 2026-02-22T20:34:19.646 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105425.log 2026-02-22T20:34:19.647 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68839.log 2026-02-22T20:34:19.647 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48151.log: /var/log/ceph/cluster2-client.mirror.105425.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117084.log 2026-02-22T20:34:19.647 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68839.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48151.log.gz 2026-02-22T20:34:19.648 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105425.log.gz 2026-02-22T20:34:19.648 INFO:teuthology.orchestra.run.trial066.stderr:gzip 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68839.log.gz -5 2026-02-22T20:34:19.648 INFO:teuthology.orchestra.run.trial066.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.109885.log 2026-02-22T20:34:19.648 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83362.log 2026-02-22T20:34:19.648 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117084.log: /var/log/ceph/cluster2-client.mirror.109885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27262.log 2026-02-22T20:34:19.648 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.83362.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109885.log.gz 2026-02-22T20:34:19.648 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117084.log.gz 2026-02-22T20:34:19.648 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126649.log 2026-02-22T20:34:19.649 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25460.log 2026-02-22T20:34:19.649 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27262.log: /var/log/ceph/cluster1-client.mirror.126649.log: 87.0% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27262.log.gz 2026-02-22T20:34:19.649 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126649.log.gz 2026-02-22T20:34:19.649 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95187.log 2026-02-22T20:34:19.649 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64836.log 2026-02-22T20:34:19.650 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25460.log: /var/log/ceph/cluster1-client.mirror.95187.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.25460.log.gz 2026-02-22T20:34:19.650 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129659.log 2026-02-22T20:34:19.651 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64836.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64836.log.gz 2026-02-22T20:34:19.651 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24251.log 2026-02-22T20:34:19.651 INFO:teuthology.orchestra.run.trial066.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.95187.log.gz 2026-02-22T20:34:19.651 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.83362.log.gz 2026-02-22T20:34:19.651 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129659.log: 83.8% -- replaced with /var/log/ceph/cluster1-client.mirror.129659.log.gz 2026-02-22T20:34:19.652 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96150.log 2026-02-22T20:34:19.652 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72245.log 2026-02-22T20:34:19.652 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24251.log: /var/log/ceph/cluster2-client.mirror.96150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97928.log 2026-02-22T20:34:19.652 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72245.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.24251.log.gz 2026-02-22T20:34:19.653 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96150.log.gz 2026-02-22T20:34:19.653 INFO:teuthology.orchestra.run.trial066.stderr:gzip 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72245.log.gz -5 2026-02-22T20:34:19.653 INFO:teuthology.orchestra.run.trial066.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.121941.log 2026-02-22T20:34:19.653 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124593.log 2026-02-22T20:34:19.653 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97928.log: /var/log/ceph/cluster2-client.mirror.121941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22949.log 2026-02-22T20:34:19.654 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124593.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121941.log.gz 2026-02-22T20:34:19.654 INFO:teuthology.orchestra.run.trial066.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124593.log.gz 2026-02-22T20:34:19.654 INFO:teuthology.orchestra.run.trial066.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97928.log.gz 2026-02-22T20:34:19.655 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125591.log 2026-02-22T20:34:19.655 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40594.log 2026-02-22T20:34:19.655 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22949.log: /var/log/ceph/cluster1-client.mirror.125591.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125591.log.gz 2026-02-22T20:34:19.656 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.22949.log.gz 2026-02-22T20:34:19.656 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.87307.log 2026-02-22T20:34:19.656 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35967.log 2026-02-22T20:34:19.656 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.40594.log: /var/log/ceph/cluster1-client.mirror.0.87307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19805.log 2026-02-22T20:34:19.656 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35967.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.40594.log.gz 2026-02-22T20:34:19.657 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35967.log.gz 2026-02-22T20:34:19.657 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70571.log 2026-02-22T20:34:19.657 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.19805.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19805.log.gz 2026-02-22T20:34:19.657 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.80678.log 2026-02-22T20:34:19.658 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103154.log 2026-02-22T20:34:19.658 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70571.log: /var/log/ceph/cluster2-client.mirror.0.80678.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70571.log.gz 2026-02-22T20:34:19.659 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106798.log 2026-02-22T20:34:19.659 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103154.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103154.log.gz 2026-02-22T20:34:19.659 INFO:teuthology.orchestra.run.trial066.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.80678.log.gz 2026-02-22T20:34:19.659 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50139.log 2026-02-22T20:34:19.660 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123021.log 2026-02-22T20:34:19.660 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106798.log: /var/log/ceph/cluster2-client.mirror.50139.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50139.log.gz 2026-02-22T20:34:19.666 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121005.log 2026-02-22T20:34:19.667 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123021.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123021.log.gz 2026-02-22T20:34:19.667 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94567.log 2026-02-22T20:34:19.668 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121005.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121005.log.gz 2026-02-22T20:34:19.668 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109650.log 2026-02-22T20:34:19.669 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94567.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94567.log.gz 2026-02-22T20:34:19.669 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82380.log 2026-02-22T20:34:19.670 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.109650.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109650.log.gz 2026-02-22T20:34:19.670 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82708.log 2026-02-22T20:34:19.671 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82380.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82380.log.gz 2026-02-22T20:34:19.671 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74942.log 2026-02-22T20:34:19.674 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82708.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104704.log.gz 2026-02-22T20:34:19.674 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63499.log 2026-02-22T20:34:19.675 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74942.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.82708.log.gz 2026-02-22T20:34:19.675 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74942.log.gz 2026-02-22T20:34:19.675 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130050.log 2026-02-22T20:34:19.675 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118753.log 2026-02-22T20:34:19.676 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63499.log: /var/log/ceph/cluster2-client.mirror.130050.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63499.log.gz 2026-02-22T20:34:19.676 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130050.log.gz 2026-02-22T20:34:19.676 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108724.log 2026-02-22T20:34:19.677 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.88030.log 2026-02-22T20:34:19.677 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.118753.log: /var/log/ceph/cluster2-client.mirror.108724.log: 88.3% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108724.log.gz 2026-02-22T20:34:19.677 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.118753.log.gz 2026-02-22T20:34:19.678 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.128189.log 2026-02-22T20:34:19.678 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.1.88030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118686.log 2026-02-22T20:34:19.682 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.0.128189.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.128189.log.gz 2026-02-22T20:34:19.683 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108495.log 2026-02-22T20:34:19.683 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118686.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118686.log.gz 2026-02-22T20:34:19.684 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118981.log 2026-02-22T20:34:19.684 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108495.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108495.log.gz 2026-02-22T20:34:19.685 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35655.log 2026-02-22T20:34:19.685 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118981.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118981.log.gz 2026-02-22T20:34:19.686 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100541.log 2026-02-22T20:34:19.689 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.35655.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35655.log.gz 2026-02-22T20:34:19.689 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128062.log 2026-02-22T20:34:19.698 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82817.log 2026-02-22T20:34:19.699 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128062.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128062.log.gz 2026-02-22T20:34:19.710 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123883.log 2026-02-22T20:34:19.714 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82817.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82817.log.gz 2026-02-22T20:34:19.715 INFO:teuthology.orchestra.run.trial066.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.88030.log.gz 2026-02-22T20:34:19.715 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39630.log 2026-02-22T20:34:19.716 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123883.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123883.log.gz 2026-02-22T20:34:19.716 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57110.log 2026-02-22T20:34:19.717 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39630.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39630.log.gz 2026-02-22T20:34:19.717 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66289.log 2026-02-22T20:34:19.718 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57110.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57110.log.gz 2026-02-22T20:34:19.718 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24286.log 2026-02-22T20:34:19.718 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66289.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66289.log.gz 2026-02-22T20:34:19.719 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124990.log 2026-02-22T20:34:19.719 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24286.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.24286.log.gz 2026-02-22T20:34:19.720 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40707.log 2026-02-22T20:34:19.720 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124990.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85982.log.gz 2026-02-22T20:34:19.721 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84585.log 2026-02-22T20:34:19.721 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40707.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124990.log.gz 2026-02-22T20:34:19.721 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40707.log.gz 2026-02-22T20:34:19.721 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128189.log 2026-02-22T20:34:19.722 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42474.log 2026-02-22T20:34:19.722 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84585.log.gz 2026-02-22T20:34:19.722 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.128189.log: 85.7% -- replaced with /var/log/ceph/cluster1-client.mirror.128189.log.gz 2026-02-22T20:34:19.722 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83512.log 2026-02-22T20:34:19.723 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42474.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42474.log.gz 2026-02-22T20:34:19.726 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87163.log 2026-02-22T20:34:19.727 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51255.log 2026-02-22T20:34:19.727 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83512.log: /var/log/ceph/cluster2-client.mirror.87163.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87163.log.gz 2026-02-22T20:34:19.727 INFO:teuthology.orchestra.run.trial066.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83512.log.gz 2026-02-22T20:34:19.728 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109182.log 2026-02-22T20:34:19.728 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46694.log 2026-02-22T20:34:19.728 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51255.log.gz 2026-02-22T20:34:19.728 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47965.log 2026-02-22T20:34:19.729 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46694.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109182.log.gz 2026-02-22T20:34:19.729 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46694.log.gz 2026-02-22T20:34:19.729 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94320.log 2026-02-22T20:34:19.729 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48930.log 2026-02-22T20:34:19.730 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47965.log: /var/log/ceph/cluster2-client.mirror.94320.log: 88.3% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.94320.log.gz 2026-02-22T20:34:19.730 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47965.log.gz 2026-02-22T20:34:19.730 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79004.log 2026-02-22T20:34:19.731 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126581.log 2026-02-22T20:34:19.731 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48930.log.gz 2026-02-22T20:34:19.731 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.20097.log 2026-02-22T20:34:19.732 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79004.log.gz 2026-02-22T20:34:19.732 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126581.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126581.log.gz 2026-02-22T20:34:19.732 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73675.log 2026-02-22T20:34:19.732 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122438.log 2026-02-22T20:34:19.733 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.1.20097.log: /var/log/ceph/cluster2-client.mirror.73675.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73675.log.gz 2026-02-22T20:34:19.746 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81826.log 2026-02-22T20:34:19.747 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122438.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122438.log.gz 2026-02-22T20:34:19.748 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131711.log 2026-02-22T20:34:19.748 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81826.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81826.log.gz 2026-02-22T20:34:19.749 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25367.log 2026-02-22T20:34:19.749 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131711.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131711.log.gz 2026-02-22T20:34:19.750 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115149.log 2026-02-22T20:34:19.750 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25367.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25367.log.gz 2026-02-22T20:34:19.750 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96115.log 2026-02-22T20:34:19.751 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115149.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115149.log.gz 2026-02-22T20:34:19.751 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32668.log 2026-02-22T20:34:19.752 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96115.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96115.log.gz 2026-02-22T20:34:19.752 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118778.log 2026-02-22T20:34:19.753 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32668.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32668.log.gz 2026-02-22T20:34:19.753 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110974.log 2026-02-22T20:34:19.755 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118778.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118778.log.gz 2026-02-22T20:34:19.755 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20596.log 2026-02-22T20:34:19.756 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110974.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.110974.log.gz 2026-02-22T20:34:19.756 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19627.log 2026-02-22T20:34:19.756 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20596.log: 54.9% -- replaced with /var/log/ceph/cluster2-client.mirror.20596.log.gz 2026-02-22T20:34:19.757 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23042.log 2026-02-22T20:34:19.757 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19627.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19627.log.gz 2026-02-22T20:34:19.758 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44834.log 2026-02-22T20:34:19.758 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23042.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.23042.log.gz 2026-02-22T20:34:19.758 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28662.log 2026-02-22T20:34:19.759 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44834.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44834.log.gz 2026-02-22T20:34:19.759 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58448.log 2026-02-22T20:34:19.760 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28662.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.28662.log.gz 2026-02-22T20:34:19.760 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122154.log 2026-02-22T20:34:19.761 INFO:teuthology.orchestra.run.trial066.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.20097.log.gz 2026-02-22T20:34:19.761 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31238.log 2026-02-22T20:34:19.762 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58448.log.gz 2026-02-22T20:34:19.762 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118721.log 2026-02-22T20:34:19.762 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31238.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122154.log.gz 2026-02-22T20:34:19.762 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31238.log.gz 2026-02-22T20:34:19.763 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82473.log 2026-02-22T20:34:19.763 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63534.log 2026-02-22T20:34:19.763 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118721.log.gz 2026-02-22T20:34:19.763 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111156.log 2026-02-22T20:34:19.764 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63534.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63534.log.gz 2026-02-22T20:34:19.764 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48895.log 2026-02-22T20:34:19.767 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111156.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82473.log.gz 2026-02-22T20:34:19.767 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69955.log 2026-02-22T20:34:19.768 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48895.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48895.log.gz 2026-02-22T20:34:19.768 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96422.log 2026-02-22T20:34:19.769 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69955.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69955.log.gz 2026-02-22T20:34:19.769 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71408.log 2026-02-22T20:34:19.770 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96422.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96422.log.gz 2026-02-22T20:34:19.770 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31145.log 2026-02-22T20:34:19.771 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71408.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71408.log.gz 2026-02-22T20:34:19.771 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56924.log 2026-02-22T20:34:19.772 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31145.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31145.log.gz 2026-02-22T20:34:19.772 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62595.log 2026-02-22T20:34:19.772 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56924.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35786.log.gz 2026-02-22T20:34:19.773 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56924.log.gz 2026-02-22T20:34:19.773 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83385.log 2026-02-22T20:34:19.773 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97269.log 2026-02-22T20:34:19.774 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62595.log: /var/log/ceph/cluster2-client.mirror.83385.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62595.log.gz 2026-02-22T20:34:19.774 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19855.log 2026-02-22T20:34:19.774 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97269.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97269.log.gz 2026-02-22T20:34:19.775 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90213.log 2026-02-22T20:34:19.775 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19855.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19855.log.gz 2026-02-22T20:34:19.775 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125254.log 2026-02-22T20:34:19.776 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90213.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83385.log.gz 2026-02-22T20:34:19.777 INFO:teuthology.orchestra.run.trial066.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90213.log.gz 2026-02-22T20:34:19.777 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113291.log 2026-02-22T20:34:19.777 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112882.log 2026-02-22T20:34:19.778 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125254.log: /var/log/ceph/cluster2-client.mirror.113291.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125254.log.gz 2026-02-22T20:34:19.778 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113291.log.gz 2026-02-22T20:34:19.778 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125299.log 2026-02-22T20:34:19.779 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59157.log 2026-02-22T20:34:19.779 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.112882.log: /var/log/ceph/cluster1-client.mirror.125299.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112882.log.gz 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125299.log.gz 2026-02-22T20:34:19.779 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.779 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80910.log 2026-02-22T20:34:19.780 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59157.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59157.log.gz 2026-02-22T20:34:19.780 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38644.log 2026-02-22T20:34:19.781 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80910.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.80910.log.gz 2026-02-22T20:34:19.781 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123834.log 2026-02-22T20:34:19.782 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38644.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38644.log.gz 2026-02-22T20:34:19.782 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127730.log 2026-02-22T20:34:19.783 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123834.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123834.log.gz 2026-02-22T20:34:19.783 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73861.log 2026-02-22T20:34:19.783 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127730.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127730.log.gz 2026-02-22T20:34:19.784 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88232.log 2026-02-22T20:34:19.784 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73861.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73861.log.gz 2026-02-22T20:34:19.785 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72989.log 2026-02-22T20:34:19.799 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92306.log 2026-02-22T20:34:19.799 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72989.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72989.log.gz 2026-02-22T20:34:19.800 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66196.log 2026-02-22T20:34:19.801 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92306.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92306.log.gz 2026-02-22T20:34:19.801 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57796.log 2026-02-22T20:34:19.802 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66196.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66196.log.gz 2026-02-22T20:34:19.802 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52379.log.gz 2026-02-22T20:34:19.802 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25495.log 2026-02-22T20:34:19.803 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26704.log 2026-02-22T20:34:19.804 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25495.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57796.log.gz 2026-02-22T20:34:19.804 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.25495.log.gz 2026-02-22T20:34:19.804 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48465.log 2026-02-22T20:34:19.804 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86774.log 2026-02-22T20:34:19.805 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26704.log: /var/log/ceph/cluster2-client.mirror.48465.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26704.log.gz 2026-02-22T20:34:19.805 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48465.log.gz 2026-02-22T20:34:19.805 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99159.log 2026-02-22T20:34:19.805 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117502.log 2026-02-22T20:34:19.805 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86774.log: /var/log/ceph/cluster2-client.mirror.99159.log: 84.7% 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85480.log.gz 2026-02-22T20:34:19.806 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.86774.log.gz 2026-02-22T20:34:19.806 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99159.log.gz 2026-02-22T20:34:19.806 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36472.log 2026-02-22T20:34:19.807 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23728.log 2026-02-22T20:34:19.807 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.36472.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117502.log.gz 2026-02-22T20:34:19.807 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99609.log 2026-02-22T20:34:19.807 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28999.log 2026-02-22T20:34:19.807 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.23728.log.gz 2026-02-22T20:34:19.808 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99609.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99609.log.gz 2026-02-22T20:34:19.808 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58541.log 2026-02-22T20:34:19.808 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126851.log 2026-02-22T20:34:19.809 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.28999.log.gz 2026-02-22T20:34:19.809 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58541.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58541.log.gz 2026-02-22T20:34:19.809 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127211.log 2026-02-22T20:34:19.809 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126851.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126851.log.gz 2026-02-22T20:34:19.809 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126784.log 2026-02-22T20:34:19.810 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127211.log: 87.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51348.log 2026-02-22T20:34:19.810 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.127211.log.gz 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36472.log.gz 2026-02-22T20:34:19.810 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.810 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32854.log 2026-02-22T20:34:19.810 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51348.log: 86.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49360.log 2026-02-22T20:34:19.810 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126784.log.gz 2026-02-22T20:34:19.810 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51348.log.gz 2026-02-22T20:34:19.811 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32854.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32854.log.gz 2026-02-22T20:34:19.811 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25181.log 2026-02-22T20:34:19.811 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25216.log 2026-02-22T20:34:19.812 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25181.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49360.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113780.log 2026-02-22T20:34:19.812 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.812 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.25181.log.gz 2026-02-22T20:34:19.812 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87249.log 2026-02-22T20:34:19.812 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.25216.log.gz 2026-02-22T20:34:19.813 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124493.log 2026-02-22T20:34:19.813 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113780.log: /var/log/ceph/cluster2-client.mirror.87249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61384.log 2026-02-22T20:34:19.813 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113780.log.gz 2026-02-22T20:34:19.813 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87249.log.gz 2026-02-22T20:34:19.814 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.20030.log 2026-02-22T20:34:19.814 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49081.log 2026-02-22T20:34:19.814 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97970.log.gz 2026-02-22T20:34:19.814 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.2.20030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28157.log 2026-02-22T20:34:19.815 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49081.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87272.log.gz 2026-02-22T20:34:19.815 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124493.log.gz 2026-02-22T20:34:19.815 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49081.log.gz 2026-02-22T20:34:19.815 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125569.log 2026-02-22T20:34:19.815 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105502.log 2026-02-22T20:34:19.816 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28157.log.gz 2026-02-22T20:34:19.816 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112989.log 2026-02-22T20:34:19.816 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125569.log.gz 2026-02-22T20:34:19.816 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.105502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92840.log 2026-02-22T20:34:19.816 INFO:teuthology.orchestra.run.trial066.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.20030.log.gz 2026-02-22T20:34:19.816 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114831.log 2026-02-22T20:34:19.816 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.112989.log: /var/log/ceph/cluster2-client.mirror.92840.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112989.log.gz 2026-02-22T20:34:19.817 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.92840.log.gz 2026-02-22T20:34:19.817 INFO:teuthology.orchestra.run.trial066.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105502.log.gz 2026-02-22T20:34:19.817 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.21880.log.gz 2026-02-22T20:34:19.820 INFO:teuthology.orchestra.run.trial066.stderr: 97.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61384.log.gz 2026-02-22T20:34:19.821 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117378.log 2026-02-22T20:34:19.821 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.129078.log 2026-02-22T20:34:19.822 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.117378.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88001.log.gz 2026-02-22T20:34:19.822 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114831.log.gz 2026-02-22T20:34:19.822 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68188.log 2026-02-22T20:34:19.822 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.3.129078.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117378.log.gz 2026-02-22T20:34:19.822 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34121.log 2026-02-22T20:34:19.823 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72559.log 2026-02-22T20:34:19.823 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34121.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68188.log.gz 2026-02-22T20:34:19.823 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109048.log 2026-02-22T20:34:19.823 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34121.log.gz 2026-02-22T20:34:19.824 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84218.log 2026-02-22T20:34:19.824 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72559.log.gz 2026-02-22T20:34:19.824 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23635.log 2026-02-22T20:34:19.824 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118415.log 2026-02-22T20:34:19.825 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79984.log 2026-02-22T20:34:19.825 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.23635.log.gz/var/log/ceph/cluster2-client.mirror.118415.log: 2026-02-22T20:34:19.825 INFO:teuthology.orchestra.run.trial066.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109048.log.gz 2026-02-22T20:34:19.825 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.20055.log 2026-02-22T20:34:19.825 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118415.log.gz 2026-02-22T20:34:19.825 INFO:teuthology.orchestra.run.trial066.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.129078.log.gz 2026-02-22T20:34:19.825 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.19983.log 2026-02-22T20:34:19.826 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.79984.log.gz/var/log/ceph/cluster1-client.mirror.3.20055.log: 2026-02-22T20:34:19.826 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75453.log 2026-02-22T20:34:19.826 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.1.19983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71815.log 2026-02-22T20:34:19.826 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91103.log 2026-02-22T20:34:19.826 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64522.log 2026-02-22T20:34:19.827 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91103.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71815.log.gz 2026-02-22T20:34:19.827 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115248.log.gz 2026-02-22T20:34:19.827 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91103.log.gz 2026-02-22T20:34:19.827 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121747.log 2026-02-22T20:34:19.828 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64522.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.19983.log.gz 2026-02-22T20:34:19.828 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27134.log 2026-02-22T20:34:19.828 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64522.log.gz 2026-02-22T20:34:19.828 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66754.log 2026-02-22T20:34:19.829 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121747.log.gz 2026-02-22T20:34:19.829 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99471.log 2026-02-22T20:34:19.829 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27134.log.gz 2026-02-22T20:34:19.829 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51127.log 2026-02-22T20:34:19.829 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.99471.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66754.log.gz 2026-02-22T20:34:19.829 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28064.log 2026-02-22T20:34:19.829 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51127.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75453.log.gz 2026-02-22T20:34:19.829 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124877.log 2026-02-22T20:34:19.831 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51127.log.gz 2026-02-22T20:34:19.831 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57482.log 2026-02-22T20:34:19.831 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28064.log: /var/log/ceph/cluster2-client.mirror.124877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27971.log 2026-02-22T20:34:19.831 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57482.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.28064.log.gz 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22798.log 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124877.log.gz 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57482.log.gz 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.99471.log.gz 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105720.log 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47624.log 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27971.log.gz 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105720.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.22798.log.gz 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63906.log 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105720.log.gz 2026-02-22T20:34:19.832 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16450.log 2026-02-22T20:34:19.833 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47624.log.gz 2026-02-22T20:34:19.833 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116384.log 2026-02-22T20:34:19.833 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.16450.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85664.log 2026-02-22T20:34:19.833 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63906.log.gz 2026-02-22T20:34:19.834 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16450.log.gz 2026-02-22T20:34:19.834 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59808.log 2026-02-22T20:34:19.834 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116384.log.gz 2026-02-22T20:34:19.834 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59471.log 2026-02-22T20:34:19.834 INFO:teuthology.orchestra.run.trial066.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85664.log.gz 2026-02-22T20:34:19.834 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102296.log 2026-02-22T20:34:19.834 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59471.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59808.log.gz 2026-02-22T20:34:19.834 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110622.log 2026-02-22T20:34:19.835 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59471.log.gz 2026-02-22T20:34:19.835 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110691.log 2026-02-22T20:34:19.835 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.102296.log.gz 2026-02-22T20:34:19.835 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120252.log 2026-02-22T20:34:19.835 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110691.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53106.log 2026-02-22T20:34:19.835 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75039.log 2026-02-22T20:34:19.835 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110691.log.gz 2026-02-22T20:34:19.835 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53106.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120252.log.gz 2026-02-22T20:34:19.836 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65545.log 2026-02-22T20:34:19.836 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53106.log.gz 2026-02-22T20:34:19.836 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114273.log 2026-02-22T20:34:19.836 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103968.log 2026-02-22T20:34:19.836 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75039.log.gz 2026-02-22T20:34:19.836 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114273.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65545.log.gz 2026-02-22T20:34:19.836 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131476.log 2026-02-22T20:34:19.837 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56234.log 2026-02-22T20:34:19.837 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131476.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103968.log.gz 2026-02-22T20:34:19.837 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86171.log 2026-02-22T20:34:19.837 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.131476.log.gz 2026-02-22T20:34:19.837 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.56234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61630.log 2026-02-22T20:34:19.837 INFO:teuthology.orchestra.run.trial066.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114273.log.gz 2026-02-22T20:34:19.837 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121468.log 2026-02-22T20:34:19.838 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86171.log: /var/log/ceph/cluster2-client.mirror.61630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64650.log 2026-02-22T20:34:19.838 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127707.log 2026-02-22T20:34:19.838 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61630.log.gz 2026-02-22T20:34:19.838 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64650.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91753.log 2026-02-22T20:34:19.838 INFO:teuthology.orchestra.run.trial066.stderr: 89.3% 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64650.log.gz 2026-02-22T20:34:19.838 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.121468.log.gz 2026-02-22T20:34:19.838 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127707.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127707.log.gz 2026-02-22T20:34:19.839 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85301.log 2026-02-22T20:34:19.839 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72524.log 2026-02-22T20:34:19.839 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91753.log: /var/log/ceph/cluster2-client.mirror.85301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73175.log 2026-02-22T20:34:19.840 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.72524.log: -- replaced with /var/log/ceph/cluster2-client.mirror.91753.log.gz 2026-02-22T20:34:19.840 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56234.log.gz 2026-02-22T20:34:19.840 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72524.log.gzgzip 2026-02-22T20:34:19.840 INFO:teuthology.orchestra.run.trial066.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.80628.log 2026-02-22T20:34:19.840 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128543.log 2026-02-22T20:34:19.841 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73175.log: /var/log/ceph/cluster1-client.admin.80628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33784.log 2026-02-22T20:34:19.841 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128543.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.80628.log.gz 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73175.log.gz 2026-02-22T20:34:19.841 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.841 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50662.log 2026-02-22T20:34:19.842 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.129659.log 2026-02-22T20:34:19.842 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86647.log.gz 2026-02-22T20:34:19.842 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33784.log: /var/log/ceph/cluster2-client.mirror.50662.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33784.log.gz 2026-02-22T20:34:19.842 INFO:teuthology.orchestra.run.trial066.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75225.log 2026-02-22T20:34:19.842 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50662.log.gz 2026-02-22T20:34:19.843 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126041.log 2026-02-22T20:34:19.843 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.1.129659.log: /var/log/ceph/cluster2-client.mirror.75225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89583.log 2026-02-22T20:34:19.843 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75225.log.gz/var/log/ceph/cluster1-client.mirror.126041.log: 2026-02-22T20:34:19.843 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126041.log.gz 2026-02-22T20:34:19.844 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100541.log.gz 2026-02-22T20:34:19.844 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126964.log 2026-02-22T20:34:19.845 INFO:teuthology.orchestra.run.trial066.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.129659.log.gz 2026-02-22T20:34:19.845 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.89583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28755.log 2026-02-22T20:34:19.845 INFO:teuthology.orchestra.run.trial066.stderr: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89583.log.gz 2026-02-22T20:34:19.845 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97136.log 2026-02-22T20:34:19.846 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126964.log: /var/log/ceph/cluster2-client.mirror.28755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86593.log 2026-02-22T20:34:19.846 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126964.log.gz 2026-02-22T20:34:19.846 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28755.log.gz 2026-02-22T20:34:19.846 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22519.log 2026-02-22T20:34:19.846 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97136.log: /var/log/ceph/cluster2-client.mirror.86593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20613.log 2026-02-22T20:34:19.846 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22519.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97136.log.gz 2026-02-22T20:34:19.846 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122970.log 2026-02-22T20:34:19.847 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20613.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.22519.log.gz 2026-02-22T20:34:19.847 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102811.log 2026-02-22T20:34:19.847 INFO:teuthology.orchestra.run.trial066.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.20613.log.gz 2026-02-22T20:34:19.847 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27448.log 2026-02-22T20:34:19.847 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102811.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84218.log.gz 2026-02-22T20:34:19.847 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71780.log 2026-02-22T20:34:19.848 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46787.log 2026-02-22T20:34:19.848 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71780.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27448.log.gz 2026-02-22T20:34:19.848 INFO:teuthology.orchestra.run.trial066.stderr: 90.7% 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71780.log.gz 2026-02-22T20:34:19.848 INFO:teuthology.orchestra.run.trial066.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102811.log.gz 2026-02-22T20:34:19.848 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.122970.log.gz 2026-02-22T20:34:19.849 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71350.log 2026-02-22T20:34:19.849 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101591.log 2026-02-22T20:34:19.849 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46787.log.gz 2026-02-22T20:34:19.849 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126874.log 2026-02-22T20:34:19.849 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71350.log.gz 2026-02-22T20:34:19.849 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69118.log 2026-02-22T20:34:19.850 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65115.log 2026-02-22T20:34:19.850 INFO:teuthology.orchestra.run.trial066.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101591.log.gz 2026-02-22T20:34:19.850 INFO:teuthology.orchestra.run.trial066.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126874.log.gz 2026-02-22T20:34:19.850 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69118.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26425.log 2026-02-22T20:34:19.850 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69118.log.gz 2026-02-22T20:34:19.850 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65115.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83659.log 2026-02-22T20:34:19.851 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26425.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65115.log.gz 2026-02-22T20:34:19.851 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65301.log 2026-02-22T20:34:19.851 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83659.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26425.log.gz 2026-02-22T20:34:19.851 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22612.log 2026-02-22T20:34:19.851 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83659.log.gz 2026-02-22T20:34:19.852 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106704.log 2026-02-22T20:34:19.852 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22612.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65301.log.gz 2026-02-22T20:34:19.852 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106887.log 2026-02-22T20:34:19.852 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.22612.log.gz 2026-02-22T20:34:19.852 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96352.log 2026-02-22T20:34:19.853 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124167.log 2026-02-22T20:34:19.853 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96352.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106704.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106887.log.gz 2026-02-22T20:34:19.854 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24623.log 2026-02-22T20:34:19.854 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.854 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96352.log.gz 2026-02-22T20:34:19.854 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124167.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124167.log.gz 2026-02-22T20:34:19.854 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103119.log 2026-02-22T20:34:19.854 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53757.log 2026-02-22T20:34:19.855 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.24623.log.gz 2026-02-22T20:34:19.855 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103119.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35839.log 2026-02-22T20:34:19.855 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103119.log.gz 2026-02-22T20:34:19.855 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.53757.log: --verbose -- /var/log/ceph/cluster2-client.mirror.72710.log 2026-02-22T20:34:19.856 INFO:teuthology.orchestra.run.trial066.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.35839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112030.log 2026-02-22T20:34:19.856 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53757.log.gz 2026-02-22T20:34:19.856 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72710.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35839.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131867.log 2026-02-22T20:34:19.856 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72710.log.gz 2026-02-22T20:34:19.856 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.856 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100499.log 2026-02-22T20:34:19.857 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.112030.log: /var/log/ceph/cluster2-client.admin.131867.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112030.log.gz 2026-02-22T20:34:19.857 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131867.log.gz 2026-02-22T20:34:19.857 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35610.log 2026-02-22T20:34:19.857 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40223.log 2026-02-22T20:34:19.857 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100499.log: /var/log/ceph/cluster1-client.mirror.35610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126019.log 2026-02-22T20:34:19.857 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40223.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41765.log 2026-02-22T20:34:19.857 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.35610.log.gz 2026-02-22T20:34:19.858 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40223.log.gz 2026-02-22T20:34:19.858 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126019.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126019.log.gz 2026-02-22T20:34:19.858 INFO:teuthology.orchestra.run.trial066.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100499.log.gz 2026-02-22T20:34:19.858 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55117.log 2026-02-22T20:34:19.859 INFO:teuthology.orchestra.run.trial066.stderr:gzip/var/log/ceph/cluster2-client.mirror.41765.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96958.log 2026-02-22T20:34:19.859 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91931.log 2026-02-22T20:34:19.859 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55117.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41765.log.gz 2026-02-22T20:34:19.860 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96958.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64987.log 2026-02-22T20:34:19.860 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55117.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96958.log.gz 2026-02-22T20:34:19.860 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:19.860 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48093.log 2026-02-22T20:34:19.860 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91931.log.gz/var/log/ceph/cluster2-client.mirror.64987.log: 2026-02-22T20:34:19.860 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85092.log 2026-02-22T20:34:19.860 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48093.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64987.log.gz 2026-02-22T20:34:19.861 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49918.log 2026-02-22T20:34:19.861 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48093.log.gz 2026-02-22T20:34:19.861 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113567.log 2026-02-22T20:34:19.861 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22763.log 2026-02-22T20:34:19.861 INFO:teuthology.orchestra.run.trial066.stderr: 88.6%/var/log/ceph/cluster1-client.mirror.113567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95167.log 2026-02-22T20:34:19.861 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49918.log.gz 2026-02-22T20:34:19.862 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22763.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113567.log.gz 2026-02-22T20:34:19.862 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.22763.log.gz 2026-02-22T20:34:19.862 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113950.log 2026-02-22T20:34:19.862 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.95167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73640.log 2026-02-22T20:34:19.863 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76904.log 2026-02-22T20:34:19.863 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113950.log.gz 2026-02-22T20:34:19.863 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73640.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73640.log.gz 2026-02-22T20:34:19.863 INFO:teuthology.orchestra.run.trial066.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95167.log.gz 2026-02-22T20:34:19.864 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102607.log 2026-02-22T20:34:19.864 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127981.log 2026-02-22T20:34:19.864 INFO:teuthology.orchestra.run.trial066.stderr: 89.1%/var/log/ceph/cluster2-client.mirror.102607.log: -- replaced with /var/log/ceph/cluster2-client.mirror.76904.log.gz 2026-02-22T20:34:19.864 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93744.log 2026-02-22T20:34:19.865 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.127981.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.127981.log.gz 2026-02-22T20:34:19.865 INFO:teuthology.orchestra.run.trial066.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102607.log.gz 2026-02-22T20:34:19.865 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39851.log 2026-02-22T20:34:19.865 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50848.log 2026-02-22T20:34:19.866 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39851.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.93744.log.gz 2026-02-22T20:34:19.866 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121397.log 2026-02-22T20:34:19.866 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39851.log.gz 2026-02-22T20:34:19.866 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50848.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50848.log.gz 2026-02-22T20:34:19.866 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131084.log 2026-02-22T20:34:19.866 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64185.log 2026-02-22T20:34:19.867 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120284.log 2026-02-22T20:34:19.867 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131084.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121397.log.gz 2026-02-22T20:34:19.867 INFO:teuthology.orchestra.run.trial066.stderr: 83.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131084.log.gz 2026-02-22T20:34:19.867 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119813.log 2026-02-22T20:34:19.867 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64185.log.gz 2026-02-22T20:34:19.868 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.120284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47159.log 2026-02-22T20:34:19.868 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.119813.log: -- replaced with /var/log/ceph/cluster1-client.mirror.120284.log.gz 2026-02-22T20:34:19.868 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52558.log 2026-02-22T20:34:19.868 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119813.log.gz 2026-02-22T20:34:19.868 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47159.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47159.log.gz 2026-02-22T20:34:19.869 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38830.log 2026-02-22T20:34:19.869 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33377.log 2026-02-22T20:34:19.870 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61145.log 2026-02-22T20:34:19.870 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38830.log: /var/log/ceph/cluster2-client.mirror.33377.log: 88.5% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33377.log.gz 2026-02-22T20:34:19.870 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38830.log.gz 2026-02-22T20:34:19.870 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61407.log 2026-02-22T20:34:19.871 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105305.log 2026-02-22T20:34:19.871 INFO:teuthology.orchestra.run.trial066.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.61407.log: -- replaced with /var/log/ceph/cluster2-client.mirror.61145.log.gz 2026-02-22T20:34:19.871 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130922.log 2026-02-22T20:34:19.872 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105305.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105305.log.gz 2026-02-22T20:34:19.872 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124686.log 2026-02-22T20:34:19.873 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.130922.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.130922.log.gz 2026-02-22T20:34:19.873 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24193.log 2026-02-22T20:34:19.874 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124686.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124686.log.gz 2026-02-22T20:34:19.874 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104197.log 2026-02-22T20:34:19.875 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24193.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.24193.log.gz 2026-02-22T20:34:19.875 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73954.log 2026-02-22T20:34:19.876 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104197.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104197.log.gz 2026-02-22T20:34:19.876 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97725.log 2026-02-22T20:34:19.877 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73954.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73954.log.gz 2026-02-22T20:34:19.877 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115060.log 2026-02-22T20:34:19.878 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.97725.log: 88.1% 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84496.log.gz 2026-02-22T20:34:19.878 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97725.log.gz 2026-02-22T20:34:19.878 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59064.log 2026-02-22T20:34:19.879 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115060.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106852.log 2026-02-22T20:34:19.879 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115060.log.gz/var/log/ceph/cluster2-client.mirror.59064.log: 2026-02-22T20:34:19.879 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59064.log.gz 2026-02-22T20:34:19.879 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97055.log 2026-02-22T20:34:19.880 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36517.log 2026-02-22T20:34:19.880 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106852.log.gz 2026-02-22T20:34:19.880 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97055.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97055.log.gz 2026-02-22T20:34:19.880 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51778.log 2026-02-22T20:34:19.881 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36517.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98364.log 2026-02-22T20:34:19.881 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51778.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51778.log.gz 2026-02-22T20:34:19.882 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112217.log 2026-02-22T20:34:19.882 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98364.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98364.log.gz 2026-02-22T20:34:19.883 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19755.log 2026-02-22T20:34:19.898 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.112217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34493.log 2026-02-22T20:34:19.899 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.19755.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19755.log.gz 2026-02-22T20:34:19.899 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31273.log 2026-02-22T20:34:19.899 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34493.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34493.log.gz 2026-02-22T20:34:19.915 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22426.log 2026-02-22T20:34:19.915 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31273.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31273.log.gz 2026-02-22T20:34:19.922 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48337.log 2026-02-22T20:34:19.923 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22426.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.22426.log.gz 2026-02-22T20:34:19.942 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79039.log 2026-02-22T20:34:19.943 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48337.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48337.log.gz 2026-02-22T20:34:19.954 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.55836.log 2026-02-22T20:34:19.955 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79039.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79039.log.gz 2026-02-22T20:34:19.966 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95142.log 2026-02-22T20:34:19.979 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.2.55836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91053.log 2026-02-22T20:34:19.979 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.95142.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95142.log.gz 2026-02-22T20:34:19.982 INFO:teuthology.orchestra.run.trial066.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106798.log.gz 2026-02-22T20:34:19.983 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124544.log 2026-02-22T20:34:19.983 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91053.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91053.log.gz 2026-02-22T20:34:19.984 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99666.log 2026-02-22T20:34:19.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124544.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124544.log.gz 2026-02-22T20:34:19.985 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116922.log 2026-02-22T20:34:19.986 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99666.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99666.log.gz 2026-02-22T20:34:19.987 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30801.log 2026-02-22T20:34:19.987 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112217.log.gz 2026-02-22T20:34:19.987 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116922.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116922.log.gz 2026-02-22T20:34:19.988 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101168.log 2026-02-22T20:34:19.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30801.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.30801.log.gz 2026-02-22T20:34:19.989 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74791.log 2026-02-22T20:34:19.989 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101168.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101168.log.gz 2026-02-22T20:34:19.990 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83827.log 2026-02-22T20:34:19.990 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74791.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74791.log.gz 2026-02-22T20:34:19.991 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105685.log 2026-02-22T20:34:19.991 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83827.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83827.log.gz 2026-02-22T20:34:19.992 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67033.log 2026-02-22T20:34:19.992 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105685.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105685.log.gz 2026-02-22T20:34:19.993 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103433.log 2026-02-22T20:34:19.993 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67033.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67033.log.gz 2026-02-22T20:34:19.994 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39944.log 2026-02-22T20:34:19.994 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103433.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103433.log.gz 2026-02-22T20:34:19.995 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59657.log 2026-02-22T20:34:19.995 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39944.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39944.log.gz 2026-02-22T20:34:19.996 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20081.log 2026-02-22T20:34:19.996 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59657.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59657.log.gz 2026-02-22T20:34:19.996 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69176.log 2026-02-22T20:34:19.997 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20081.log: 86.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20081.log.gz 2026-02-22T20:34:19.998 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129750.log 2026-02-22T20:34:19.998 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69176.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69176.log.gz 2026-02-22T20:34:19.998 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97525.log 2026-02-22T20:34:19.999 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129750.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129750.log.gz 2026-02-22T20:34:19.999 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22205.log 2026-02-22T20:34:20.000 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97525.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.97525.log.gz 2026-02-22T20:34:20.000 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78199.log 2026-02-22T20:34:20.001 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22205.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.22205.log.gz 2026-02-22T20:34:20.001 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129565.log 2026-02-22T20:34:20.002 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78199.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78199.log.gz 2026-02-22T20:34:20.002 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128314.log 2026-02-22T20:34:20.003 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129565.log: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129565.log.gz 2026-02-22T20:34:20.003 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131685.log 2026-02-22T20:34:20.004 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.128314.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.128314.log.gz 2026-02-22T20:34:20.004 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55024.log 2026-02-22T20:34:20.005 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131685.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131685.log.gz 2026-02-22T20:34:20.005 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68060.log 2026-02-22T20:34:20.005 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55024.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55024.log.gz 2026-02-22T20:34:20.014 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50604.log 2026-02-22T20:34:20.015 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68060.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68060.log.gz 2026-02-22T20:34:20.015 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120074.log 2026-02-22T20:34:20.017 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50604.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50604.log.gz 2026-02-22T20:34:20.017 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.89337.log.gz 2026-02-22T20:34:20.017 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82751.log 2026-02-22T20:34:20.017 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92487.log 2026-02-22T20:34:20.017 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120074.log.gz 2026-02-22T20:34:20.019 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28813.log 2026-02-22T20:34:20.019 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110622.log.gz/var/log/ceph/cluster2-client.mirror.92487.log: 2026-02-22T20:34:20.019 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92487.log.gz 2026-02-22T20:34:20.020 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123741.log 2026-02-22T20:34:20.020 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126896.log 2026-02-22T20:34:20.020 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123741.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.28813.log.gz 2026-02-22T20:34:20.020 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90546.log 2026-02-22T20:34:20.020 INFO:teuthology.orchestra.run.trial066.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123741.log.gz 2026-02-22T20:34:20.020 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126896.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126896.log.gz 2026-02-22T20:34:20.024 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81015.log 2026-02-22T20:34:20.024 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34342.log 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.81015.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.82751.log.gz 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.90546.log.gz 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81015.log.gz 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68932.log 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90658.log 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68932.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34342.log.gz 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45450.log 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90658.log: 88.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106949.log 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68932.log.gz 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90658.log.gz 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45450.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25774.log 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45450.log.gz 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83319.log 2026-02-22T20:34:20.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25774.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106949.log.gz 2026-02-22T20:34:20.026 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25774.log.gz 2026-02-22T20:34:20.026 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114216.log 2026-02-22T20:34:20.026 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.83319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76519.log 2026-02-22T20:34:20.026 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96546.log 2026-02-22T20:34:20.026 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114216.log.gz/var/log/ceph/cluster2-client.mirror.76519.log: 2026-02-22T20:34:20.026 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76519.log.gz 2026-02-22T20:34:20.027 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.83319.log.gz 2026-02-22T20:34:20.027 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90735.log 2026-02-22T20:34:20.027 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48709.log 2026-02-22T20:34:20.029 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.90735.log: -- replaced with /var/log/ceph/cluster2-client.mirror.96546.log.gz 2026-02-22T20:34:20.029 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53350.log 2026-02-22T20:34:20.029 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84934.log 2026-02-22T20:34:20.029 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53350.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48709.log.gz 2026-02-22T20:34:20.029 INFO:teuthology.orchestra.run.trial066.stderr: 92.4% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53350.log.gz 2026-02-22T20:34:20.029 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.90735.log.gz 2026-02-22T20:34:20.029 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34586.log 2026-02-22T20:34:20.030 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112025.log 2026-02-22T20:34:20.030 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56959.log 2026-02-22T20:34:20.031 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34586.log.gz 2026-02-22T20:34:20.031 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.112025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104298.log 2026-02-22T20:34:20.031 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56959.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56959.log.gz 2026-02-22T20:34:20.031 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37024.log 2026-02-22T20:34:20.031 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104298.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104298.log.gz 2026-02-22T20:34:20.032 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52214.log 2026-02-22T20:34:20.032 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37024.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37024.log.gz 2026-02-22T20:34:20.032 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32354.log 2026-02-22T20:34:20.033 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52214.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112025.log.gz 2026-02-22T20:34:20.033 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52214.log.gz 2026-02-22T20:34:20.033 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69641.log 2026-02-22T20:34:20.034 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53815.log 2026-02-22T20:34:20.034 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32354.log: /var/log/ceph/cluster2-client.mirror.69641.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32354.log.gz 2026-02-22T20:34:20.034 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69641.log.gz 2026-02-22T20:34:20.034 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35312.log 2026-02-22T20:34:20.035 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37214.log 2026-02-22T20:34:20.035 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53815.log: /var/log/ceph/cluster2-client.mirror.35312.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35312.log.gz 2026-02-22T20:34:20.036 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53815.log.gz 2026-02-22T20:34:20.036 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62037.log 2026-02-22T20:34:20.036 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37214.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37214.log.gz 2026-02-22T20:34:20.036 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73361.log 2026-02-22T20:34:20.037 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62037.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62037.log.gz 2026-02-22T20:34:20.037 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29929.log 2026-02-22T20:34:20.038 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73361.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73361.log.gz 2026-02-22T20:34:20.038 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123041.log 2026-02-22T20:34:20.039 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29929.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29929.log.gz 2026-02-22T20:34:20.039 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101203.log 2026-02-22T20:34:20.041 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123041.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123041.log.gz 2026-02-22T20:34:20.041 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83342.log 2026-02-22T20:34:20.042 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101203.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101203.log.gz 2026-02-22T20:34:20.042 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126289.log 2026-02-22T20:34:20.045 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83342.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83342.log.gz 2026-02-22T20:34:20.046 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83449.log 2026-02-22T20:34:20.046 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38946.log 2026-02-22T20:34:20.047 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126289.log: /var/log/ceph/cluster2-client.mirror.83449.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126289.log.gz 2026-02-22T20:34:20.048 INFO:teuthology.orchestra.run.trial066.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131063.log 2026-02-22T20:34:20.048 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.83449.log.gz 2026-02-22T20:34:20.048 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93891.log 2026-02-22T20:34:20.048 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131063.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38946.log.gz 2026-02-22T20:34:20.048 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38272.log 2026-02-22T20:34:20.049 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.93891.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131063.log.gz 2026-02-22T20:34:20.049 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85508.log 2026-02-22T20:34:20.049 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38272.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38272.log.gz 2026-02-22T20:34:20.050 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37900.log 2026-02-22T20:34:20.050 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85508.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85508.log.gz 2026-02-22T20:34:20.051 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45915.log 2026-02-22T20:34:20.051 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37900.log: 95.0% 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93891.log.gz 2026-02-22T20:34:20.051 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.37900.log.gz 2026-02-22T20:34:20.051 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116061.log 2026-02-22T20:34:20.052 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105207.log 2026-02-22T20:34:20.052 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45915.log: /var/log/ceph/cluster1-client.mirror.116061.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45915.log.gz 2026-02-22T20:34:20.052 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62960.log 2026-02-22T20:34:20.053 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105207.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116061.log.gz 2026-02-22T20:34:20.053 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105207.log.gz 2026-02-22T20:34:20.053 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35049.log 2026-02-22T20:34:20.054 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79529.log 2026-02-22T20:34:20.054 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35049.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35049.log.gz 2026-02-22T20:34:20.057 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62960.log.gz 2026-02-22T20:34:20.057 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84473.log 2026-02-22T20:34:20.058 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79529.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79529.log.gz 2026-02-22T20:34:20.058 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87432.log 2026-02-22T20:34:20.058 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130840.log 2026-02-22T20:34:20.059 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87432.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84473.log.gz 2026-02-22T20:34:20.059 INFO:teuthology.orchestra.run.trial066.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87432.log.gz 2026-02-22T20:34:20.059 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54280.log 2026-02-22T20:34:20.060 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125501.log 2026-02-22T20:34:20.060 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130840.log: /var/log/ceph/cluster2-client.mirror.54280.log: 87.7% 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130840.log.gz 2026-02-22T20:34:20.060 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54280.log.gz 2026-02-22T20:34:20.060 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128339.log 2026-02-22T20:34:20.061 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122950.log 2026-02-22T20:34:20.061 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125501.log: /var/log/ceph/cluster2-client.mirror.128339.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125501.log.gz 2026-02-22T20:34:20.061 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120217.log 2026-02-22T20:34:20.062 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122950.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128339.log.gz 2026-02-22T20:34:20.062 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.122950.log.gz 2026-02-22T20:34:20.062 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131307.log 2026-02-22T20:34:20.062 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82794.log 2026-02-22T20:34:20.063 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120217.log: /var/log/ceph/cluster2-client.mirror.131307.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120217.log.gz 2026-02-22T20:34:20.063 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131307.log.gz 2026-02-22T20:34:20.063 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128414.log 2026-02-22T20:34:20.064 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62409.log 2026-02-22T20:34:20.064 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82794.log: /var/log/ceph/cluster2-client.mirror.128414.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.128414.log.gz 2026-02-22T20:34:20.067 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.82794.log.gz 2026-02-22T20:34:20.067 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124280.log 2026-02-22T20:34:20.067 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62409.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83947.log 2026-02-22T20:34:20.068 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62409.log.gz 2026-02-22T20:34:20.068 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111789.log 2026-02-22T20:34:20.069 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83947.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83947.log.gz 2026-02-22T20:34:20.069 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90715.log 2026-02-22T20:34:20.069 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111789.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124280.log.gz 2026-02-22T20:34:20.069 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120535.log 2026-02-22T20:34:20.070 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90715.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111789.log.gz 2026-02-22T20:34:20.070 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98624.log 2026-02-22T20:34:20.071 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120535.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120535.log.gz 2026-02-22T20:34:20.071 INFO:teuthology.orchestra.run.trial066.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.90715.log.gz 2026-02-22T20:34:20.072 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127369.log 2026-02-22T20:34:20.072 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104003.log 2026-02-22T20:34:20.072 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98624.log.gz 2026-02-22T20:34:20.073 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127369.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119458.log.gz 2026-02-22T20:34:20.073 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58971.log 2026-02-22T20:34:20.073 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127369.log.gz 2026-02-22T20:34:20.073 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54001.log 2026-02-22T20:34:20.073 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104003.log.gz 2026-02-22T20:34:20.074 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55987.log 2026-02-22T20:34:20.074 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99356.log 2026-02-22T20:34:20.074 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58971.log.gz 2026-02-22T20:34:20.074 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54001.log: /var/log/ceph/cluster2-client.mirror.55987.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54001.log.gz 88.7% 2026-02-22T20:34:20.074 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55987.log.gzgzip 2026-02-22T20:34:20.074 INFO:teuthology.orchestra.run.trial066.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127877.log 2026-02-22T20:34:20.074 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74977.log 2026-02-22T20:34:20.074 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99356.log.gz 2026-02-22T20:34:20.075 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.127877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68095.log 2026-02-22T20:34:20.075 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74977.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74977.log.gz 2026-02-22T20:34:20.075 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55582.log 2026-02-22T20:34:20.076 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68095.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68095.log.gz 2026-02-22T20:34:20.076 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92875.log 2026-02-22T20:34:20.076 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55582.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55582.log.gz 2026-02-22T20:34:20.077 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130098.log 2026-02-22T20:34:20.077 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92875.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.92875.log.gz 2026-02-22T20:34:20.078 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.75628.log 2026-02-22T20:34:20.079 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130098.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63068.log.gz 2026-02-22T20:34:20.093 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129342.log.gz 2026-02-22T20:34:20.093 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117942.log 2026-02-22T20:34:20.093 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.0.75628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22391.log 2026-02-22T20:34:20.094 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49395.log 2026-02-22T20:34:20.094 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117942.log: /var/log/ceph/cluster2-client.mirror.22391.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117942.log.gz 2026-02-22T20:34:20.094 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.22391.log.gz 2026-02-22T20:34:20.095 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111086.log 2026-02-22T20:34:20.095 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54315.log 2026-02-22T20:34:20.096 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49395.log: /var/log/ceph/cluster1-client.mirror.111086.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49395.log.gz 2026-02-22T20:34:20.096 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65766.log 2026-02-22T20:34:20.096 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54315.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111086.log.gz 2026-02-22T20:34:20.097 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54315.log.gz 2026-02-22T20:34:20.097 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.93891.log 2026-02-22T20:34:20.097 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96635.log 2026-02-22T20:34:20.098 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65766.log.gz 2026-02-22T20:34:20.098 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.0.93891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101924.log 2026-02-22T20:34:20.098 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96635.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96635.log.gz 2026-02-22T20:34:20.110 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69083.log 2026-02-22T20:34:20.111 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101924.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101924.log.gz 2026-02-22T20:34:20.111 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33563.log 2026-02-22T20:34:20.112 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69083.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69083.log.gz 2026-02-22T20:34:20.112 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111992.log 2026-02-22T20:34:20.113 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33563.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33563.log.gz 2026-02-22T20:34:20.113 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93547.log 2026-02-22T20:34:20.114 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111992.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111992.log.gz 2026-02-22T20:34:20.114 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106558.log 2026-02-22T20:34:20.115 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93547.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93547.log.gz 2026-02-22T20:34:20.115 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16533.log 2026-02-22T20:34:20.116 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106558.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106558.log.gz 2026-02-22T20:34:20.116 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97560.log 2026-02-22T20:34:20.117 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.16533.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16533.log.gz 2026-02-22T20:34:20.117 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72187.log 2026-02-22T20:34:20.118 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97560.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.97560.log.gz 2026-02-22T20:34:20.118 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63871.log 2026-02-22T20:34:20.118 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72187.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72187.log.gz 2026-02-22T20:34:20.119 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126266.log 2026-02-22T20:34:20.119 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63871.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63871.log.gz 2026-02-22T20:34:20.120 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75602.log 2026-02-22T20:34:20.120 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126266.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126266.log.gz 2026-02-22T20:34:20.121 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80216.log 2026-02-22T20:34:20.138 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54780.log 2026-02-22T20:34:20.139 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80216.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80216.log.gz 2026-02-22T20:34:20.150 INFO:teuthology.orchestra.run.trial066.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.93891.log.gz 2026-02-22T20:34:20.151 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69920.log 2026-02-22T20:34:20.151 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54780.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54780.log.gz 2026-02-22T20:34:20.152 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92711.log 2026-02-22T20:34:20.153 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69920.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69920.log.gz 2026-02-22T20:34:20.153 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125099.log 2026-02-22T20:34:20.153 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92711.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.92711.log.gz 2026-02-22T20:34:20.154 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32947.log 2026-02-22T20:34:20.154 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125099.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.125099.log.gz 2026-02-22T20:34:20.155 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46380.log 2026-02-22T20:34:20.155 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32947.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32947.log.gz 2026-02-22T20:34:20.156 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86967.log 2026-02-22T20:34:20.156 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46380.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46380.log.gz 2026-02-22T20:34:20.157 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106647.log 2026-02-22T20:34:20.158 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86967.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86967.log.gz 2026-02-22T20:34:20.158 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125019.log 2026-02-22T20:34:20.158 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106647.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86171.log.gz 2026-02-22T20:34:20.158 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106647.log.gz 2026-02-22T20:34:20.159 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110268.log 2026-02-22T20:34:20.159 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.125019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53943.log 2026-02-22T20:34:20.160 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110268.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125019.log.gz 2026-02-22T20:34:20.161 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110268.log.gz 2026-02-22T20:34:20.161 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51627.log 2026-02-22T20:34:20.161 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44555.log 2026-02-22T20:34:20.161 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53943.log.gz 2026-02-22T20:34:20.164 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63720.log 2026-02-22T20:34:20.164 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44555.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51627.log.gz 2026-02-22T20:34:20.165 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44555.log.gz 2026-02-22T20:34:20.165 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102719.log 2026-02-22T20:34:20.165 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123976.log 2026-02-22T20:34:20.165 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63720.log: /var/log/ceph/cluster2-client.mirror.102719.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.102719.log.gz 2026-02-22T20:34:20.165 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63720.log.gz 2026-02-22T20:34:20.165 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79424.log 2026-02-22T20:34:20.165 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123976.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123976.log.gz 2026-02-22T20:34:20.177 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111156.log.gz 2026-02-22T20:34:20.177 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94624.log 2026-02-22T20:34:20.177 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46845.log 2026-02-22T20:34:20.178 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79424.log.gz/var/log/ceph/cluster2-client.mirror.94624.log: 2026-02-22T20:34:20.178 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127301.log 2026-02-22T20:34:20.178 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91455.log 2026-02-22T20:34:20.178 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46845.log: /var/log/ceph/cluster1-client.mirror.127301.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46845.log.gz 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127301.log.gz 2026-02-22T20:34:20.178 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.178 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65824.log 2026-02-22T20:34:20.178 INFO:teuthology.orchestra.run.trial066.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94624.log.gz 2026-02-22T20:34:20.178 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60273.log 2026-02-22T20:34:20.179 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122551.log 2026-02-22T20:34:20.179 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65824.log: /var/log/ceph/cluster2-client.mirror.60273.log: 89.0% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60273.log.gz 2026-02-22T20:34:20.179 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65824.log.gz 2026-02-22T20:34:20.179 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120128.log 2026-02-22T20:34:20.179 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130737.log 2026-02-22T20:34:20.179 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120128.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120128.log.gz 2026-02-22T20:34:20.179 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73826.log 2026-02-22T20:34:20.179 INFO:teuthology.orchestra.run.trial066.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122551.log.gz 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83533.log 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73826.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130737.log.gz 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73826.log.gz 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82622.log 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59192.log 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83533.log: /var/log/ceph/cluster1-client.mirror.82622.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83533.log.gz 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41637.log 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85092.log.gz 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115872.log 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59192.log.gz 2026-02-22T20:34:20.188 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68618.log 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115872.log: 97.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91455.log.gz 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41637.log.gz 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115872.log.gz 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41451.log 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76169.log 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr: 89.0%/var/log/ceph/cluster2-client.mirror.41451.log: -- replaced with /var/log/ceph/cluster2-client.mirror.68618.log.gz 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125344.log 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41451.log.gz 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr: 93.1%/var/log/ceph/cluster2-client.mirror.76169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98329.log 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.82622.log.gz 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124615.log 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76169.log.gz 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.125344.log.gz 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20323.log 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98329.log.gz 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39013.log 2026-02-22T20:34:20.189 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124615.log: /var/log/ceph/cluster1-client.admin.20323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131022.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124615.log.gz 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr: 70.8% -- replaced with /var/log/ceph/cluster1-client.admin.20323.log.gz 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23914.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr: 88.5%/var/log/ceph/cluster2-client.mirror.131022.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39013.log.gz 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128110.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131022.log.gz 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51069.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.23914.log.gz 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68711.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128110.log: /var/log/ceph/cluster2-client.mirror.51069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121207.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82202.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.51069.log.gz 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68711.log: /var/log/ceph/cluster2-client.mirror.121207.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121207.log.gz 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68711.log.gz 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80467.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96724.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.80467.log: 89.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74698.log 2026-02-22T20:34:20.190 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.82202.log.gz 2026-02-22T20:34:20.191 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96724.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96724.log.gz 2026-02-22T20:34:20.191 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82989.log 2026-02-22T20:34:20.191 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128684.log 2026-02-22T20:34:20.193 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74698.log: /var/log/ceph/cluster2-client.mirror.82989.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74698.log.gz 2026-02-22T20:34:20.193 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20732.log 2026-02-22T20:34:20.193 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.128684.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128684.log.gz 2026-02-22T20:34:20.193 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98445.log 2026-02-22T20:34:20.193 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.80467.log.gz 2026-02-22T20:34:20.193 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20732.log: 56.3% -- replaced with /var/log/ceph/cluster2-client.mirror.20732.log.gz 2026-02-22T20:34:20.193 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109310.log 2026-02-22T20:34:20.193 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94107.log 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98445.log: /var/log/ceph/cluster2-client.mirror.109310.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.98445.log.gz 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109310.log.gz 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128022.log 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82989.log.gz 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58878.log 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94107.log: /var/log/ceph/cluster2-client.mirror.128022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83299.log 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58878.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94107.log.gz 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76064.log 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58878.log.gz 2026-02-22T20:34:20.194 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83299.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128022.log.gz 2026-02-22T20:34:20.195 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83299.log.gz 2026-02-22T20:34:20.196 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121242.log 2026-02-22T20:34:20.196 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109275.log 2026-02-22T20:34:20.197 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119141.log 2026-02-22T20:34:20.197 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121242.log: 89.1%/var/log/ceph/cluster2-client.mirror.109275.log: -- replaced with /var/log/ceph/cluster2-client.mirror.76064.log.gz 2026-02-22T20:34:20.197 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109275.log.gz 2026-02-22T20:34:20.197 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121242.log.gz 2026-02-22T20:34:20.197 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107356.log 2026-02-22T20:34:20.197 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97444.log 2026-02-22T20:34:20.198 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.107356.log: -- replaced with /var/log/ceph/cluster2-client.mirror.119141.log.gz 2026-02-22T20:34:20.198 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21852.log 2026-02-22T20:34:20.198 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107356.log.gz 2026-02-22T20:34:20.198 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97444.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97444.log.gz 2026-02-22T20:34:20.199 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71687.log 2026-02-22T20:34:20.199 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.21852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.34857.log 2026-02-22T20:34:20.199 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93582.log 2026-02-22T20:34:20.199 INFO:teuthology.orchestra.run.trial066.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.21852.log.gz 2026-02-22T20:34:20.199 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71687.log.gz 2026-02-22T20:34:20.204 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.1.34857.log: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.34857.log.gz 2026-02-22T20:34:20.204 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55934.log 2026-02-22T20:34:20.205 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20408.log 2026-02-22T20:34:20.205 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102242.log 2026-02-22T20:34:20.205 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55934.log: 88.3%/var/log/ceph/cluster1-client.mirror.20408.log: -- replaced with /var/log/ceph/cluster2-client.mirror.93582.log.gz 2026-02-22T20:34:20.206 INFO:teuthology.orchestra.run.trial066.stderr: 64.7% -- replaced with /var/log/ceph/cluster1-client.mirror.20408.log.gz 2026-02-22T20:34:20.206 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55934.log.gz 2026-02-22T20:34:20.206 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47310.log 2026-02-22T20:34:20.207 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63348.log 2026-02-22T20:34:20.207 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.102242.log.gz 2026-02-22T20:34:20.207 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74884.log 2026-02-22T20:34:20.207 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61537.log 2026-02-22T20:34:20.207 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47310.log.gz 2026-02-22T20:34:20.209 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63348.log.gz 2026-02-22T20:34:20.209 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49953.log 2026-02-22T20:34:20.209 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74884.log: /var/log/ceph/cluster2-client.mirror.61537.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74884.log.gz 2026-02-22T20:34:20.209 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61537.log.gz 2026-02-22T20:34:20.209 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2026-02-22T20:34:20.209 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99437.log 2026-02-22T20:34:20.209 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster2-osd.2.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49953.log.gz 2026-02-22T20:34:20.209 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65952.log 2026-02-22T20:34:20.209 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107275.log 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58169.log 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.107275.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99437.log.gz 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107275.log.gz 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88232.log.gz 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65952.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65952.log.gz 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40095.log 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67591.log 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102831.log 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.40095.log: -- replaced with /var/log/ceph/cluster2-client.mirror.58169.log.gz 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67591.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40095.log.gz 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67591.log.gz 2026-02-22T20:34:20.212 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23077.log 2026-02-22T20:34:20.213 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.102831.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.102831.log.gz 2026-02-22T20:34:20.213 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110233.log 2026-02-22T20:34:20.214 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109997.log 2026-02-22T20:34:20.214 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23077.log: /var/log/ceph/cluster2-client.mirror.110233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46566.log 2026-02-22T20:34:20.215 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.23077.log.gz 2026-02-22T20:34:20.215 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.109997.log: -- /var/log/ceph/cluster2-client.mirror.67498.log 2026-02-22T20:34:20.215 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110233.log.gz 2026-02-22T20:34:20.215 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46566.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46566.log.gz 2026-02-22T20:34:20.215 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86990.log 2026-02-22T20:34:20.216 INFO:teuthology.orchestra.run.trial066.stderr: 92.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32296.log 2026-02-22T20:34:20.216 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.109997.log.gz 2026-02-22T20:34:20.216 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67498.log: /var/log/ceph/cluster2-client.mirror.86990.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67498.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19349.log 2026-02-22T20:34:20.216 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.217 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32296.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32296.log.gz 2026-02-22T20:34:20.217 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55431.log 2026-02-22T20:34:20.217 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19349.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128543.log.gz 2026-02-22T20:34:20.217 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42009.log 2026-02-22T20:34:20.218 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19349.log.gz 2026-02-22T20:34:20.219 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70606.log 2026-02-22T20:34:20.219 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104035.log 2026-02-22T20:34:20.219 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55431.log.gz 2026-02-22T20:34:20.219 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70606.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42009.log.gz 2026-02-22T20:34:20.219 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117680.log 2026-02-22T20:34:20.219 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70606.log.gz 2026-02-22T20:34:20.219 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102118.log 2026-02-22T20:34:20.220 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.104035.log: /var/log/ceph/cluster2-client.mirror.117680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52179.log 2026-02-22T20:34:20.220 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104035.log.gz 2026-02-22T20:34:20.220 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102118.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117680.log.gz 2026-02-22T20:34:20.220 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31424.log 2026-02-22T20:34:20.220 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.102118.log.gz 2026-02-22T20:34:20.221 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97903.log 2026-02-22T20:34:20.221 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31424.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52179.log.gz 2026-02-22T20:34:20.221 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31424.log.gz 2026-02-22T20:34:20.221 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73082.log 2026-02-22T20:34:20.222 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124706.log 2026-02-22T20:34:20.222 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.97903.log: /var/log/ceph/cluster2-client.mirror.73082.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2026-02-22T20:34:20.222 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73082.log.gz 2026-02-22T20:34:20.222 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.97903.log.gz 2026-02-22T20:34:20.222 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124706.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.99471.log 2026-02-22T20:34:20.223 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113815.log 2026-02-22T20:34:20.223 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-osd.1.log: /var/log/ceph/cluster2-client.mirror.3.99471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38086.log 2026-02-22T20:34:20.223 INFO:teuthology.orchestra.run.trial066.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124706.log.gz 2026-02-22T20:34:20.223 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56257.log 2026-02-22T20:34:20.224 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38086.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38086.log.gz 2026-02-22T20:34:20.224 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39572.log 2026-02-22T20:34:20.228 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56257.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56257.log.gz 2026-02-22T20:34:20.228 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81931.log 2026-02-22T20:34:20.229 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39572.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39572.log.gz 2026-02-22T20:34:20.229 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114459.log 2026-02-22T20:34:20.230 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81931.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81931.log.gz 2026-02-22T20:34:20.230 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.129750.log 2026-02-22T20:34:20.231 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113815.log: /var/log/ceph/cluster2-client.mirror.114459.log: 88.2% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114459.log.gz 2026-02-22T20:34:20.231 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.113815.log.gz 2026-02-22T20:34:20.231 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110403.log 2026-02-22T20:34:20.238 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.1.129750.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.129750.log.gz 2026-02-22T20:34:20.239 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34028.log 2026-02-22T20:34:20.239 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122673.log 2026-02-22T20:34:20.239 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110403.log.gz 2026-02-22T20:34:20.240 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34028.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34028.log.gz 2026-02-22T20:34:20.240 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86207.log 2026-02-22T20:34:20.241 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122673.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122673.log.gz 2026-02-22T20:34:20.241 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19315.log 2026-02-22T20:34:20.242 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86207.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86207.log.gz 2026-02-22T20:34:20.242 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87455.log 2026-02-22T20:34:20.242 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19315.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19315.log.gz 2026-02-22T20:34:20.243 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19880.log 2026-02-22T20:34:20.251 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110167.log 2026-02-22T20:34:20.251 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19880.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19880.log.gz 2026-02-22T20:34:20.252 INFO:teuthology.orchestra.run.trial066.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.99471.log.gz 2026-02-22T20:34:20.254 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84608.log.gz 2026-02-22T20:34:20.271 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53257.log 2026-02-22T20:34:20.272 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130818.log 2026-02-22T20:34:20.272 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110167.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110167.log.gz/var/log/ceph/cluster2-client.mirror.53257.log: 2026-02-22T20:34:20.272 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53257.log.gz 2026-02-22T20:34:20.272 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109920.log 2026-02-22T20:34:20.272 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128393.log 2026-02-22T20:34:20.273 INFO:teuthology.orchestra.run.trial066.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.109920.log: -- replaced with /var/log/ceph/cluster2-client.mirror.130818.log.gz 2026-02-22T20:34:20.273 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109920.log.gz 2026-02-22T20:34:20.273 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26797.log 2026-02-22T20:34:20.273 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.128393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89314.log 2026-02-22T20:34:20.273 INFO:teuthology.orchestra.run.trial066.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.128393.log.gz 2026-02-22T20:34:20.274 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93304.log 2026-02-22T20:34:20.274 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89314.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26797.log.gz 2026-02-22T20:34:20.274 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.89314.log.gz 2026-02-22T20:34:20.274 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77884.log 2026-02-22T20:34:20.275 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104605.log 2026-02-22T20:34:20.275 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93304.log: 88.2%/var/log/ceph/cluster2-client.mirror.77884.log: -- replaced with /var/log/ceph/cluster2-client.mirror.93304.log.gz 2026-02-22T20:34:20.276 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77884.log.gz 2026-02-22T20:34:20.276 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103793.log 2026-02-22T20:34:20.276 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104605.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104605.log.gz 2026-02-22T20:34:20.282 INFO:teuthology.orchestra.run.trial066.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.127877.log.gz 2026-02-22T20:34:20.282 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125974.log 2026-02-22T20:34:20.283 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91885.log 2026-02-22T20:34:20.283 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125974.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103793.log.gz 2026-02-22T20:34:20.283 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28848.log 2026-02-22T20:34:20.283 INFO:teuthology.orchestra.run.trial066.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125974.log.gz 2026-02-22T20:34:20.284 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41079.log 2026-02-22T20:34:20.284 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91885.log: /var/log/ceph/cluster2-client.mirror.28848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116696.log 2026-02-22T20:34:20.284 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28848.log.gz 2026-02-22T20:34:20.284 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41079.log: -- replaced with /var/log/ceph/cluster2-client.mirror.91885.log.gz 2026-02-22T20:34:20.284 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41079.log.gz 2026-02-22T20:34:20.285 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62002.log 2026-02-22T20:34:20.285 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97792.log 2026-02-22T20:34:20.285 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123283.log 2026-02-22T20:34:20.285 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62002.log.gz 2026-02-22T20:34:20.286 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123692.log 2026-02-22T20:34:20.286 INFO:teuthology.orchestra.run.trial066.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75602.log.gz 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116696.log.gz 2026-02-22T20:34:20.286 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.287 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123283.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.123283.log.gz 2026-02-22T20:34:20.287 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68153.log 2026-02-22T20:34:20.288 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122247.log 2026-02-22T20:34:20.288 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68153.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123692.log.gz 2026-02-22T20:34:20.288 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52978.log 2026-02-22T20:34:20.288 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122247.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68153.log.gz 2026-02-22T20:34:20.288 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61274.log 2026-02-22T20:34:20.288 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122247.log.gz 2026-02-22T20:34:20.289 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52978.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52978.log.gz 2026-02-22T20:34:20.291 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110438.log 2026-02-22T20:34:20.291 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.61274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19930.log 2026-02-22T20:34:20.291 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110438.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61274.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130758.log 2026-02-22T20:34:20.291 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.admin.19930.log: -- replaced with /var/log/ceph/cluster2-client.mirror.110438.log.gz 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19930.log.gz 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41486.log 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69397.log 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41486.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130758.log.gz 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68653.log 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69397.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41486.log.gz 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70141.log 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68653.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69397.log.gz 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82731.log 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68653.log.gz 2026-02-22T20:34:20.292 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70141.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70141.log.gz 2026-02-22T20:34:20.293 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70885.log 2026-02-22T20:34:20.293 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69025.log 2026-02-22T20:34:20.294 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70978.log 2026-02-22T20:34:20.294 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69025.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70885.log.gz 2026-02-22T20:34:20.294 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69025.log.gz 2026-02-22T20:34:20.294 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56080.log 2026-02-22T20:34:20.294 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70978.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70978.log.gz 2026-02-22T20:34:20.295 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63406.log 2026-02-22T20:34:20.295 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43962.log 2026-02-22T20:34:20.295 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63406.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56080.log.gz 2026-02-22T20:34:20.295 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63406.log.gz 2026-02-22T20:34:20.295 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82731.log.gz 2026-02-22T20:34:20.296 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112195.log 2026-02-22T20:34:20.296 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130256.log 2026-02-22T20:34:20.296 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.112195.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43962.log.gz 2026-02-22T20:34:20.296 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72652.log 2026-02-22T20:34:20.297 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130256.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131103.log 2026-02-22T20:34:20.297 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72652.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130256.log.gz 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72652.log.gz 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113167.log 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr: 92.3%/var/log/ceph/cluster2-client.mirror.131103.log: -- replaced with /var/log/ceph/cluster1-client.mirror.112195.log.gz 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64615.log 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.131103.log.gz 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91567.log 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.64615.log: -- replaced with /var/log/ceph/cluster2-client.mirror.113167.log.gz 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82237.log 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64615.log.gz 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.80769.log 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91567.log.gz 2026-02-22T20:34:20.299 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100402.log 2026-02-22T20:34:20.300 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82237.log.gz 2026-02-22T20:34:20.300 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.1.80769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32761.log 2026-02-22T20:34:20.300 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80379.log 2026-02-22T20:34:20.300 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.32761.log: -- replaced with /var/log/ceph/cluster2-client.mirror.100402.log.gz 2026-02-22T20:34:20.300 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96457.log 2026-02-22T20:34:20.300 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.32761.log.gz 2026-02-22T20:34:20.301 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.80379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110488.log 2026-02-22T20:34:20.302 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96457.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.80379.log.gz 2026-02-22T20:34:20.302 INFO:teuthology.orchestra.run.trial066.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.80769.log.gz 2026-02-22T20:34:20.302 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96457.log.gz 2026-02-22T20:34:20.302 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53850.log 2026-02-22T20:34:20.302 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57517.log 2026-02-22T20:34:20.302 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.53850.log: -- replaced with /var/log/ceph/cluster2-client.mirror.110488.log.gz 2026-02-22T20:34:20.302 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43160.log 2026-02-22T20:34:20.302 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53850.log.gz/var/log/ceph/cluster2-client.mirror.57517.log: 2026-02-22T20:34:20.302 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58506.log 2026-02-22T20:34:20.303 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52558.log.gz 2026-02-22T20:34:20.303 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57517.log.gz 2026-02-22T20:34:20.303 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23170.log 2026-02-22T20:34:20.304 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43160.log.gz 2026-02-22T20:34:20.304 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108375.log 2026-02-22T20:34:20.304 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58506.log.gz 2026-02-22T20:34:20.304 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117211.log 2026-02-22T20:34:20.304 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108375.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.23170.log.gz 2026-02-22T20:34:20.304 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82148.log 2026-02-22T20:34:20.304 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127753.log 2026-02-22T20:34:20.304 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82148.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117211.log.gz 2026-02-22T20:34:20.305 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82148.log.gz 2026-02-22T20:34:20.305 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99686.log 2026-02-22T20:34:20.305 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32203.log 2026-02-22T20:34:20.305 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127753.log.gz 2026-02-22T20:34:20.305 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.99686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53071.log 2026-02-22T20:34:20.305 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32203.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32203.log.gz 2026-02-22T20:34:20.306 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114552.log 2026-02-22T20:34:20.306 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53071.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99686.log.gz 2026-02-22T20:34:20.307 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53071.log.gz 2026-02-22T20:34:20.307 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49116.log 2026-02-22T20:34:20.307 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114552.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114552.log.gz 2026-02-22T20:34:20.307 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65638.log 2026-02-22T20:34:20.308 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31831.log 2026-02-22T20:34:20.308 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65638.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49116.log.gz 2026-02-22T20:34:20.308 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65638.log.gz 2026-02-22T20:34:20.308 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116671.log 2026-02-22T20:34:20.309 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107078.log 2026-02-22T20:34:20.309 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31831.log.gz/var/log/ceph/cluster1-client.mirror.116671.log: 2026-02-22T20:34:20.309 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116671.log.gz 2026-02-22T20:34:20.309 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125794.log 2026-02-22T20:34:20.310 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.107078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123554.log 2026-02-22T20:34:20.310 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107078.log.gz 2026-02-22T20:34:20.311 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125794.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125794.log.gz 2026-02-22T20:34:20.311 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112777.log 2026-02-22T20:34:20.312 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123554.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123554.log.gz 2026-02-22T20:34:20.312 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81438.log 2026-02-22T20:34:20.313 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.112777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101786.log 2026-02-22T20:34:20.313 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112777.log.gz/var/log/ceph/cluster2-client.mirror.81438.log: 2026-02-22T20:34:20.313 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81438.log.gz 2026-02-22T20:34:20.313 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25646.log 2026-02-22T20:34:20.313 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.101786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100937.log 2026-02-22T20:34:20.314 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25646.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25646.log.gz 2026-02-22T20:34:20.314 INFO:teuthology.orchestra.run.trial066.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101786.log.gz 2026-02-22T20:34:20.314 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81527.log 2026-02-22T20:34:20.315 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95920.log 2026-02-22T20:34:20.315 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81527.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81527.log.gz 2026-02-22T20:34:20.326 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28285.log 2026-02-22T20:34:20.327 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.95920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40002.log 2026-02-22T20:34:20.327 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28285.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95920.log.gz 2026-02-22T20:34:20.327 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28285.log.gz 2026-02-22T20:34:20.328 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74105.log 2026-02-22T20:34:20.328 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41858.log 2026-02-22T20:34:20.328 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40002.log.gz 2026-02-22T20:34:20.329 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61723.log 2026-02-22T20:34:20.329 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74105.log.gz 2026-02-22T20:34:20.329 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41858.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41858.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104637.log 2026-02-22T20:34:20.329 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.330 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129169.log 2026-02-22T20:34:20.330 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61723.log.gz 2026-02-22T20:34:20.330 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.104637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54687.log 2026-02-22T20:34:20.330 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.104637.log.gz 2026-02-22T20:34:20.331 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98202.log 2026-02-22T20:34:20.331 INFO:teuthology.orchestra.run.trial066.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129169.log.gz 2026-02-22T20:34:20.332 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39107.log 2026-02-22T20:34:20.332 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54687.log.gz 2026-02-22T20:34:20.332 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98202.log: gzip 88.3% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77534.log 2026-02-22T20:34:20.333 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.98202.log.gz 2026-02-22T20:34:20.333 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126356.log 2026-02-22T20:34:20.333 INFO:teuthology.orchestra.run.trial066.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39107.log.gz/var/log/ceph/cluster2-client.mirror.77534.log: 2026-02-22T20:34:20.334 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77534.log.gz 2026-02-22T20:34:20.334 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82415.log 2026-02-22T20:34:20.334 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90121.log 2026-02-22T20:34:20.334 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126356.log.gz/var/log/ceph/cluster2-client.mirror.82415.log: 2026-02-22T20:34:20.334 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82415.log.gz 2026-02-22T20:34:20.334 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82923.log 2026-02-22T20:34:20.334 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49488.log 2026-02-22T20:34:20.335 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82923.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90121.log.gz 2026-02-22T20:34:20.335 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114293.log 2026-02-22T20:34:20.335 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49488.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49488.log.gz 2026-02-22T20:34:20.336 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20459.log 2026-02-22T20:34:20.337 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.114293.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114293.log.gz 2026-02-22T20:34:20.337 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123583.log 2026-02-22T20:34:20.338 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20459.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.82923.log.gz 2026-02-22T20:34:20.338 INFO:teuthology.orchestra.run.trial066.stderr: 56.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20459.log.gz 2026-02-22T20:34:20.338 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43811.log 2026-02-22T20:34:20.339 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123583.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.123583.log.gz 2026-02-22T20:34:20.339 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129990.log 2026-02-22T20:34:20.340 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43811.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61407.log.gz 2026-02-22T20:34:20.340 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43811.log.gz 2026-02-22T20:34:20.341 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55524.log 2026-02-22T20:34:20.341 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129990.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93103.log 2026-02-22T20:34:20.341 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55524.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129990.log.gz 2026-02-22T20:34:20.341 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55524.log.gz 2026-02-22T20:34:20.342 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32540.log 2026-02-22T20:34:20.342 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93103.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80251.log 2026-02-22T20:34:20.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32540.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32540.log.gz 2026-02-22T20:34:20.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121326.log 2026-02-22T20:34:20.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80251.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80251.log.gz 2026-02-22T20:34:20.354 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100317.log 2026-02-22T20:34:20.355 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.121326.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121326.log.gz 2026-02-22T20:34:20.355 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19549.log 2026-02-22T20:34:20.356 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100317.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100317.log.gz 2026-02-22T20:34:20.374 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121097.log 2026-02-22T20:34:20.375 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83849.log 2026-02-22T20:34:20.375 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19549.log.gz 2026-02-22T20:34:20.376 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101808.log 2026-02-22T20:34:20.376 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83849.log: 88.3% 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83849.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.121097.log.gz 2026-02-22T20:34:20.376 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.376 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43311.log 2026-02-22T20:34:20.377 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39258.log 2026-02-22T20:34:20.377 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43311.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43311.log.gz 2026-02-22T20:34:20.378 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82579.log 2026-02-22T20:34:20.378 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39258.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39258.log.gz 2026-02-22T20:34:20.379 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24100.log 2026-02-22T20:34:20.383 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82579.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82579.log.gz 2026-02-22T20:34:20.383 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125479.log 2026-02-22T20:34:20.384 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24100.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.24100.log.gz 2026-02-22T20:34:20.384 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125411.log 2026-02-22T20:34:20.384 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125479.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125479.log.gz 2026-02-22T20:34:20.385 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128767.log 2026-02-22T20:34:20.385 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125411.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125411.log.gz 2026-02-22T20:34:20.386 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19905.log 2026-02-22T20:34:20.386 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128767.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128767.log.gz 2026-02-22T20:34:20.386 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24437.log 2026-02-22T20:34:20.387 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19905.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19905.log.gz 2026-02-22T20:34:20.387 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85355.log 2026-02-22T20:34:20.388 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24437.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24437.log.gz 2026-02-22T20:34:20.388 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130277.log 2026-02-22T20:34:20.389 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85355.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85355.log.gz 2026-02-22T20:34:20.389 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121139.log 2026-02-22T20:34:20.390 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130277.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130277.log.gz 2026-02-22T20:34:20.390 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28343.log 2026-02-22T20:34:20.398 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119239.log 2026-02-22T20:34:20.399 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28343.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.28343.log.gz 2026-02-22T20:34:20.407 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108375.log.gz 2026-02-22T20:34:20.410 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101048.log 2026-02-22T20:34:20.411 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119239.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119239.log.gz 2026-02-22T20:34:20.411 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83806.log 2026-02-22T20:34:20.416 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101048.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40481.log.gz 2026-02-22T20:34:20.417 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60029.log 2026-02-22T20:34:20.418 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83806.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83806.log.gz 2026-02-22T20:34:20.418 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60866.log 2026-02-22T20:34:20.419 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60029.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60029.log.gz 2026-02-22T20:34:20.419 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130072.log 2026-02-22T20:34:20.419 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60866.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60866.log.gz 2026-02-22T20:34:20.420 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129316.log 2026-02-22T20:34:20.420 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130072.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130072.log.gz 2026-02-22T20:34:20.421 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115653.log 2026-02-22T20:34:20.421 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129316.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129316.log.gz 2026-02-22T20:34:20.422 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63144.log 2026-02-22T20:34:20.422 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115653.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115653.log.gz 2026-02-22T20:34:20.423 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127824.log 2026-02-22T20:34:20.423 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.63144.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.63144.log.gz 2026-02-22T20:34:20.424 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50941.log 2026-02-22T20:34:20.424 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127824.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127824.log.gz 2026-02-22T20:34:20.425 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122580.log 2026-02-22T20:34:20.425 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50941.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50941.log.gz 2026-02-22T20:34:20.425 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71629.log 2026-02-22T20:34:20.426 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122580.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.122580.log.gz 2026-02-22T20:34:20.426 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52530.log 2026-02-22T20:34:20.426 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71629.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71629.log.gz 2026-02-22T20:34:20.427 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91321.log 2026-02-22T20:34:20.427 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.52530.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.52530.log.gz 2026-02-22T20:34:20.428 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55396.log 2026-02-22T20:34:20.428 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91321.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91321.log.gz 2026-02-22T20:34:20.428 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68990.log 2026-02-22T20:34:20.429 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55396.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55396.log.gz 2026-02-22T20:34:20.429 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45578.log 2026-02-22T20:34:20.430 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68990.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68990.log.gz 2026-02-22T20:34:20.430 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60831.log 2026-02-22T20:34:20.430 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45578.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45578.log.gz 2026-02-22T20:34:20.431 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20545.log 2026-02-22T20:34:20.431 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60831.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60831.log.gz 2026-02-22T20:34:20.432 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128370.log 2026-02-22T20:34:20.432 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.20545.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.admin.20545.log.gz 2026-02-22T20:34:20.433 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84977.log 2026-02-22T20:34:20.433 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.128370.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.128370.log.gz 2026-02-22T20:34:20.433 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73024.log 2026-02-22T20:34:20.433 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84977.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84977.log.gz 2026-02-22T20:34:20.434 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49023.log 2026-02-22T20:34:20.434 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73024.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73024.log.gz 2026-02-22T20:34:20.435 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104162.log 2026-02-22T20:34:20.435 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49023.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49023.log.gz 2026-02-22T20:34:20.435 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127279.log 2026-02-22T20:34:20.436 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104162.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104162.log.gz 2026-02-22T20:34:20.436 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112917.log 2026-02-22T20:34:20.436 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127279.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127279.log.gz 2026-02-22T20:34:20.437 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40373.log 2026-02-22T20:34:20.437 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.112917.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112917.log.gz 2026-02-22T20:34:20.438 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23321.log 2026-02-22T20:34:20.441 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40373.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40373.log.gz 2026-02-22T20:34:20.441 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62637.log 2026-02-22T20:34:20.442 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23321.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.23321.log.gz 2026-02-22T20:34:20.442 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114866.log 2026-02-22T20:34:20.443 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62637.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62637.log.gz 2026-02-22T20:34:20.443 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113592.log 2026-02-22T20:34:20.443 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114866.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114866.log.gz 2026-02-22T20:34:20.444 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99929.log 2026-02-22T20:34:20.445 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113592.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113592.log.gz 2026-02-22T20:34:20.445 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52314.log 2026-02-22T20:34:20.446 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99929.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99929.log.gz 2026-02-22T20:34:20.446 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50883.log 2026-02-22T20:34:20.450 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52314.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52314.log.gz 2026-02-22T20:34:20.450 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34857.log 2026-02-22T20:34:20.450 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50883.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50883.log.gz 2026-02-22T20:34:20.451 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125186.log 2026-02-22T20:34:20.451 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34857.log: 83.8% -- replaced with /var/log/ceph/cluster2-client.mirror.34857.log.gz 2026-02-22T20:34:20.452 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94942.log 2026-02-22T20:34:20.452 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125186.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125186.log.gz 2026-02-22T20:34:20.452 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110019.log 2026-02-22T20:34:20.455 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94942.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94942.log.gz 2026-02-22T20:34:20.455 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126086.log 2026-02-22T20:34:20.462 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110019.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86990.log.gz 2026-02-22T20:34:20.466 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31924.log 2026-02-22T20:34:20.467 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126086.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126086.log.gz 2026-02-22T20:34:20.468 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56866.log 2026-02-22T20:34:20.468 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31924.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31924.log.gz 2026-02-22T20:34:20.468 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72838.log 2026-02-22T20:34:20.469 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56866.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56866.log.gz 2026-02-22T20:34:20.469 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70106.log 2026-02-22T20:34:20.469 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72838.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72838.log.gz 2026-02-22T20:34:20.470 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33284.log 2026-02-22T20:34:20.470 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70106.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70106.log.gz 2026-02-22T20:34:20.471 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123654.log 2026-02-22T20:34:20.471 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33284.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33284.log.gz 2026-02-22T20:34:20.471 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46008.log 2026-02-22T20:34:20.472 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123654.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123654.log.gz 2026-02-22T20:34:20.473 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79844.log 2026-02-22T20:34:20.473 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46008.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46008.log.gz 2026-02-22T20:34:20.474 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96061.log 2026-02-22T20:34:20.474 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79844.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79844.log.gz 2026-02-22T20:34:20.474 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107693.log 2026-02-22T20:34:20.474 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96061.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96061.log.gz 2026-02-22T20:34:20.475 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24751.log 2026-02-22T20:34:20.477 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.107693.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107693.log.gz 2026-02-22T20:34:20.478 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45206.log 2026-02-22T20:34:20.478 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24751.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.24751.log.gz 2026-02-22T20:34:20.479 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81314.log 2026-02-22T20:34:20.479 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45206.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45206.log.gz 2026-02-22T20:34:20.479 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57761.log 2026-02-22T20:34:20.480 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81314.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81314.log.gz 2026-02-22T20:34:20.481 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123625.log 2026-02-22T20:34:20.481 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57761.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57761.log.gz 2026-02-22T20:34:20.481 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98283.log 2026-02-22T20:34:20.483 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123625.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123625.log.gz 2026-02-22T20:34:20.483 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131376.log 2026-02-22T20:34:20.484 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98283.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98283.log.gz 2026-02-22T20:34:20.485 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30990.log 2026-02-22T20:34:20.485 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.131376.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131376.log.gz 2026-02-22T20:34:20.485 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77674.log 2026-02-22T20:34:20.486 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30990.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30990.log.gz 2026-02-22T20:34:20.486 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35727.log 2026-02-22T20:34:20.487 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77674.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77674.log.gz 2026-02-22T20:34:20.487 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125389.log 2026-02-22T20:34:20.488 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.35727.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.35727.log.gz 2026-02-22T20:34:20.488 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54373.log 2026-02-22T20:34:20.488 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125389.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125389.log.gz 2026-02-22T20:34:20.489 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67723.log 2026-02-22T20:34:20.489 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54373.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54373.log.gz 2026-02-22T20:34:20.490 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79144.log 2026-02-22T20:34:20.490 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67723.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67723.log.gz 2026-02-22T20:34:20.491 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104865.log 2026-02-22T20:34:20.491 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79144.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79144.log.gz 2026-02-22T20:34:20.492 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71722.log 2026-02-22T20:34:20.492 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104865.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104865.log.gz 2026-02-22T20:34:20.492 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62374.log 2026-02-22T20:34:20.493 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71722.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71722.log.gz 2026-02-22T20:34:20.493 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122409.log 2026-02-22T20:34:20.494 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62374.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62374.log.gz 2026-02-22T20:34:20.494 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80656.log 2026-02-22T20:34:20.496 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122409.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122409.log.gz 2026-02-22T20:34:20.496 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29278.log 2026-02-22T20:34:20.503 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80656.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84740.log 2026-02-22T20:34:20.515 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78514.log 2026-02-22T20:34:20.515 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29278.log.gz 2026-02-22T20:34:20.541 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84740.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.121139.log.gz 2026-02-22T20:34:20.541 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43869.log 2026-02-22T20:34:20.542 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78514.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78514.log.gz 2026-02-22T20:34:20.542 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131633.log 2026-02-22T20:34:20.543 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43869.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43869.log.gz 2026-02-22T20:34:20.544 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81349.log 2026-02-22T20:34:20.544 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131633.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131633.log.gz 2026-02-22T20:34:20.544 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36121.log 2026-02-22T20:34:20.545 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81349.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81349.log.gz 2026-02-22T20:34:20.545 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114494.log 2026-02-22T20:34:20.546 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36121.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36121.log.gz 2026-02-22T20:34:20.546 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22670.log 2026-02-22T20:34:20.547 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114494.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114494.log.gz 2026-02-22T20:34:20.547 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50755.log 2026-02-22T20:34:20.548 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22670.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.22670.log.gz 2026-02-22T20:34:20.548 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24809.log 2026-02-22T20:34:20.549 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50755.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50755.log.gz 2026-02-22T20:34:20.549 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93242.log 2026-02-22T20:34:20.550 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24809.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24809.log.gz 2026-02-22T20:34:20.550 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68525.log 2026-02-22T20:34:20.551 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93242.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93242.log.gz 2026-02-22T20:34:20.551 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31459.log 2026-02-22T20:34:20.552 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68525.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68525.log.gz 2026-02-22T20:34:20.552 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96689.log 2026-02-22T20:34:20.552 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31459.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31459.log.gz 2026-02-22T20:34:20.553 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129606.log 2026-02-22T20:34:20.553 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96689.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96689.log.gz 2026-02-22T20:34:20.554 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129895.log 2026-02-22T20:34:20.554 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129606.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129606.log.gz 2026-02-22T20:34:20.555 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92012.log 2026-02-22T20:34:20.557 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129895.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129895.log.gz 2026-02-22T20:34:20.557 INFO:teuthology.orchestra.run.trial066.stderr: 95.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67219.log 2026-02-22T20:34:20.557 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.21426.log.gz 2026-02-22T20:34:20.557 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92012.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92012.log.gz 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125546.log 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62281.log 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67219.log: /var/log/ceph/cluster1-client.mirror.125546.log: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130098.log.gz 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67219.log.gz 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85301.log.gz 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125546.log.gz 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.130420.log 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62281.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75167.log 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62281.log.gz 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.1.130420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54129.log 2026-02-22T20:34:20.560 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75167.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75167.log.gz 2026-02-22T20:34:20.562 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120039.log 2026-02-22T20:34:20.562 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54129.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54129.log.gz 2026-02-22T20:34:20.562 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32261.log 2026-02-22T20:34:20.562 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120039.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.1.130420.log.gz 2026-02-22T20:34:20.562 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120039.log.gz 2026-02-22T20:34:20.562 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125065.log 2026-02-22T20:34:20.562 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37865.log 2026-02-22T20:34:20.562 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32261.log.gz 2026-02-22T20:34:20.563 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.125065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78794.log 2026-02-22T20:34:20.563 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37865.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37865.log.gz 2026-02-22T20:34:20.563 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119950.log 2026-02-22T20:34:20.563 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78794.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78794.log.gz 2026-02-22T20:34:20.563 INFO:teuthology.orchestra.run.trial066.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125065.log.gz 2026-02-22T20:34:20.564 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126221.log 2026-02-22T20:34:20.564 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91356.log 2026-02-22T20:34:20.564 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119950.log.gz 2026-02-22T20:34:20.564 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25681.log 2026-02-22T20:34:20.564 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126221.log.gz 2026-02-22T20:34:20.565 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91356.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.91356.log.gz 2026-02-22T20:34:20.565 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75260.log 2026-02-22T20:34:20.565 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25681.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51313.log 2026-02-22T20:34:20.565 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25681.log.gz 2026-02-22T20:34:20.566 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124331.log 2026-02-22T20:34:20.566 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75260.log.gz 2026-02-22T20:34:20.566 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51313.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51313.log.gz 2026-02-22T20:34:20.566 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.118011.log 2026-02-22T20:34:20.567 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109506.log 2026-02-22T20:34:20.567 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124331.log.gz 2026-02-22T20:34:20.568 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.2.118011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122509.log 2026-02-22T20:34:20.568 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109506.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109506.log.gz 2026-02-22T20:34:20.568 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123163.log 2026-02-22T20:34:20.568 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122509.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122509.log.gz 2026-02-22T20:34:20.569 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91701.log 2026-02-22T20:34:20.569 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123163.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123163.log.gz 2026-02-22T20:34:20.569 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26518.log 2026-02-22T20:34:20.570 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91701.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100937.log.gz 2026-02-22T20:34:20.575 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36517.log.gz 2026-02-22T20:34:20.576 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81966.log 2026-02-22T20:34:20.576 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129048.log 2026-02-22T20:34:20.576 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105122.log 2026-02-22T20:34:20.576 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81966.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26518.log.gz 2026-02-22T20:34:20.576 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129048.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81966.log.gz 2026-02-22T20:34:20.577 INFO:teuthology.orchestra.run.trial066.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129048.log.gz 2026-02-22T20:34:20.577 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68339.log 2026-02-22T20:34:20.578 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88870.log 2026-02-22T20:34:20.578 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68339.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105122.log.gz 2026-02-22T20:34:20.578 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37586.log 2026-02-22T20:34:20.579 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40393.log 2026-02-22T20:34:20.579 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68339.log.gz 2026-02-22T20:34:20.579 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88870.log: /var/log/ceph/cluster2-client.mirror.37586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20391.log 2026-02-22T20:34:20.579 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37586.log.gz 2026-02-22T20:34:20.582 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.40393.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.118011.log.gz 2026-02-22T20:34:20.582 INFO:teuthology.orchestra.run.trial066.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40393.log.gz 2026-02-22T20:34:20.584 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124473.log 2026-02-22T20:34:20.584 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22856.log 2026-02-22T20:34:20.584 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20391.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20391.log.gz 2026-02-22T20:34:20.584 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41951.log 2026-02-22T20:34:20.584 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124473.log.gz 2026-02-22T20:34:20.584 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61489.log 2026-02-22T20:34:20.584 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22856.log: /var/log/ceph/cluster2-client.mirror.41951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131789.log 2026-02-22T20:34:20.585 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.22856.log.gz 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41951.log.gz 2026-02-22T20:34:20.585 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.585 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77359.log 2026-02-22T20:34:20.585 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131789.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131789.log.gz 2026-02-22T20:34:20.585 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74384.log 2026-02-22T20:34:20.586 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77359.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77359.log.gz 2026-02-22T20:34:20.586 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110920.log 2026-02-22T20:34:20.586 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74384.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74384.log.gz 2026-02-22T20:34:20.587 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82113.log 2026-02-22T20:34:20.587 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110920.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110920.log.gz 2026-02-22T20:34:20.587 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28941.log 2026-02-22T20:34:20.588 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82113.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.82113.log.gz 2026-02-22T20:34:20.598 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51534.log 2026-02-22T20:34:20.599 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83276.log 2026-02-22T20:34:20.599 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28941.log.gz 2026-02-22T20:34:20.600 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125231.log 2026-02-22T20:34:20.600 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.83276.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51534.log.gz 2026-02-22T20:34:20.600 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52291.log 2026-02-22T20:34:20.601 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125231.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125231.log.gz 2026-02-22T20:34:20.601 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70013.log 2026-02-22T20:34:20.603 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.52291.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.83276.log.gz 2026-02-22T20:34:20.603 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60366.log 2026-02-22T20:34:20.604 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70013.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70013.log.gz 2026-02-22T20:34:20.604 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48744.log 2026-02-22T20:34:20.605 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60366.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60366.log.gz 2026-02-22T20:34:20.605 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52291.log.gz 2026-02-22T20:34:20.606 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116248.log 2026-02-22T20:34:20.606 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118516.log 2026-02-22T20:34:20.606 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48744.log.gz 2026-02-22T20:34:20.607 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31517.log 2026-02-22T20:34:20.607 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116248.log.gz 2026-02-22T20:34:20.608 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118516.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118516.log.gz 2026-02-22T20:34:20.608 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73210.log 2026-02-22T20:34:20.608 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31517.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43218.log 2026-02-22T20:34:20.609 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73210.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31517.log.gz 2026-02-22T20:34:20.609 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73210.log.gz 2026-02-22T20:34:20.609 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107321.log 2026-02-22T20:34:20.609 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35347.log 2026-02-22T20:34:20.609 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43218.log: /var/log/ceph/cluster2-client.mirror.107321.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43218.log.gz 2026-02-22T20:34:20.609 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107321.log.gz 2026-02-22T20:34:20.610 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33935.log 2026-02-22T20:34:20.610 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127900.log 2026-02-22T20:34:20.610 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35347.log: /var/log/ceph/cluster2-client.mirror.33935.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35347.log.gz 2026-02-22T20:34:20.611 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33935.log.gz 2026-02-22T20:34:20.612 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57017.log 2026-02-22T20:34:20.612 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43404.log 2026-02-22T20:34:20.612 INFO:teuthology.orchestra.run.trial066.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127900.log.gz 2026-02-22T20:34:20.612 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57017.log: 88.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97948.log 2026-02-22T20:34:20.612 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57017.log.gz 2026-02-22T20:34:20.613 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89830.log 2026-02-22T20:34:20.613 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43404.log.gz 2026-02-22T20:34:20.613 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.97948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105390.log 2026-02-22T20:34:20.614 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89830.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97948.log.gz 2026-02-22T20:34:20.614 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114721.log 2026-02-22T20:34:20.615 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105390.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105390.log.gz 2026-02-22T20:34:20.615 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89129.log 2026-02-22T20:34:20.616 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114721.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114721.log.gz 2026-02-22T20:34:20.617 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106108.log 2026-02-22T20:34:20.626 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63220.log 2026-02-22T20:34:20.627 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106108.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106108.log.gz 2026-02-22T20:34:20.630 INFO:teuthology.orchestra.run.trial066.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.110019.log.gz 2026-02-22T20:34:20.630 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63313.log 2026-02-22T20:34:20.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63220.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63220.log.gz 2026-02-22T20:34:20.631 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123467.log 2026-02-22T20:34:20.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63313.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63313.log.gz 2026-02-22T20:34:20.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65487.log 2026-02-22T20:34:20.633 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123467.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123467.log.gz 2026-02-22T20:34:20.634 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25588.log 2026-02-22T20:34:20.634 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65487.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65487.log.gz 2026-02-22T20:34:20.635 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31796.log 2026-02-22T20:34:20.635 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25588.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25588.log.gz 2026-02-22T20:34:20.636 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2026-02-22T20:34:20.636 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31796.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31796.log.gz 2026-02-22T20:34:20.636 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55803.log 2026-02-22T20:34:20.638 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86593.log.gz 2026-02-22T20:34:20.638 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40625.log 2026-02-22T20:34:20.639 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55803.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55803.log.gz 2026-02-22T20:34:20.639 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121983.log 2026-02-22T20:34:20.640 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.40625.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.40625.log.gz 2026-02-22T20:34:20.640 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28906.log 2026-02-22T20:34:20.640 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster2.log.gz 2026-02-22T20:34:20.641 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92665.log 2026-02-22T20:34:20.641 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28906.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.28906.log.gz 2026-02-22T20:34:20.642 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66475.log 2026-02-22T20:34:20.642 INFO:teuthology.orchestra.run.trial066.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.121983.log.gz 2026-02-22T20:34:20.643 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128476.log 2026-02-22T20:34:20.643 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.66475.log: -- replaced with /var/log/ceph/cluster2-client.mirror.92665.log.gz 2026-02-22T20:34:20.643 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66475.log.gz 2026-02-22T20:34:20.643 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64057.log 2026-02-22T20:34:20.643 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128476.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128476.log.gz 2026-02-22T20:34:20.644 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92452.log 2026-02-22T20:34:20.644 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37307.log 2026-02-22T20:34:20.644 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64057.log: /var/log/ceph/cluster2-client.mirror.92452.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64057.log.gz 2026-02-22T20:34:20.644 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92452.log.gz 2026-02-22T20:34:20.644 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129482.log 2026-02-22T20:34:20.646 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32482.log 2026-02-22T20:34:20.646 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37307.log: /var/log/ceph/cluster1-client.mirror.129482.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.129482.log.gz 88.5% 2026-02-22T20:34:20.646 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.37307.log.gz 2026-02-22T20:34:20.646 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117298.log 2026-02-22T20:34:20.646 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57890.log 2026-02-22T20:34:20.646 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32482.log: /var/log/ceph/cluster2-client.mirror.117298.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32482.log.gz 2026-02-22T20:34:20.646 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117298.log.gz 2026-02-22T20:34:20.647 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80656.log.gz 2026-02-22T20:34:20.647 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123812.log 2026-02-22T20:34:20.648 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113380.log 2026-02-22T20:34:20.649 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123812.log: 88.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.20155.log 2026-02-22T20:34:20.649 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57890.log.gz 2026-02-22T20:34:20.649 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72059.log 2026-02-22T20:34:20.649 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.123812.log.gz 2026-02-22T20:34:20.649 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.3.20155.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113380.log.gz 2026-02-22T20:34:20.663 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90044.log 2026-02-22T20:34:20.663 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69862.log 2026-02-22T20:34:20.663 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90044.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72059.log.gz 2026-02-22T20:34:20.664 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120662.log 2026-02-22T20:34:20.664 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90044.log.gz 2026-02-22T20:34:20.665 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63441.log 2026-02-22T20:34:20.665 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69862.log.gz 2026-02-22T20:34:20.665 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50104.log 2026-02-22T20:34:20.665 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120662.log.gz 2026-02-22T20:34:20.665 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63441.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63441.log.gz 2026-02-22T20:34:20.665 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42509.log 2026-02-22T20:34:20.667 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124919.log 2026-02-22T20:34:20.667 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50104.log.gz/var/log/ceph/cluster2-client.mirror.42509.log: 2026-02-22T20:34:20.667 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42509.log.gz 2026-02-22T20:34:20.667 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29092.log 2026-02-22T20:34:20.667 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126536.log 2026-02-22T20:34:20.667 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29092.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29092.log.gz 2026-02-22T20:34:20.667 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93466.log 2026-02-22T20:34:20.669 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126536.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126536.log.gz 2026-02-22T20:34:20.669 INFO:teuthology.orchestra.run.trial066.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124919.log.gz 2026-02-22T20:34:20.669 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129969.log 2026-02-22T20:34:20.669 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128287.log 2026-02-22T20:34:20.670 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93466.log: /var/log/ceph/cluster2-client.mirror.129969.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93466.log.gz 2026-02-22T20:34:20.670 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129969.log.gz 2026-02-22T20:34:20.670 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81050.log 2026-02-22T20:34:20.670 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89562.log 2026-02-22T20:34:20.670 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.128287.log: /var/log/ceph/cluster2-client.mirror.81050.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.128287.log.gz 2026-02-22T20:34:20.670 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81050.log.gz 2026-02-22T20:34:20.670 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.84218.log 2026-02-22T20:34:20.671 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70850.log 2026-02-22T20:34:20.671 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.89562.log: /var/log/ceph/cluster1-client.mirror.1.84218.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89562.log.gz 2026-02-22T20:34:20.671 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109618.log 2026-02-22T20:34:20.671 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70850.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70850.log.gz 2026-02-22T20:34:20.672 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45636.log 2026-02-22T20:34:20.672 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109618.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109618.log.gz 2026-02-22T20:34:20.672 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122034.log 2026-02-22T20:34:20.672 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45636.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45636.log.gz 2026-02-22T20:34:20.673 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27878.log 2026-02-22T20:34:20.673 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122034.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122034.log.gz 2026-02-22T20:34:20.673 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63255.log 2026-02-22T20:34:20.674 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27878.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27878.log.gz 2026-02-22T20:34:20.674 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90334.log 2026-02-22T20:34:20.675 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63255.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63255.log.gz 2026-02-22T20:34:20.675 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26146.log 2026-02-22T20:34:20.676 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90334.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101808.log.gz 2026-02-22T20:34:20.676 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90334.log.gz 2026-02-22T20:34:20.676 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35501.log 2026-02-22T20:34:20.677 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43776.log 2026-02-22T20:34:20.677 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26146.log.gz 2026-02-22T20:34:20.677 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20754.log 2026-02-22T20:34:20.677 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35501.log.gz 2026-02-22T20:34:20.678 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27727.log 2026-02-22T20:34:20.678 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43776.log: /var/log/ceph/cluster2-client.mirror.20754.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43776.log.gz 2026-02-22T20:34:20.678 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123212.log 2026-02-22T20:34:20.678 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.20754.log.gz 2026-02-22T20:34:20.678 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109952.log 2026-02-22T20:34:20.680 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27727.log: /var/log/ceph/cluster2-client.mirror.123212.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27727.log.gz 2026-02-22T20:34:20.681 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123212.log.gz 2026-02-22T20:34:20.681 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42660.log 2026-02-22T20:34:20.681 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.109952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50197.log 2026-02-22T20:34:20.681 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109952.log.gz 2026-02-22T20:34:20.681 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42660.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42660.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87186.log 2026-02-22T20:34:20.681 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.681 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19243.log 2026-02-22T20:34:20.681 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50197.log.gz 2026-02-22T20:34:20.682 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41300.log 2026-02-22T20:34:20.682 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19243.log: 80.8% -- replaced with /var/log/ceph/cluster2-client.admin.19243.log.gz 2026-02-22T20:34:20.682 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131328.log 2026-02-22T20:34:20.682 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41300.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41300.log.gz 2026-02-22T20:34:20.682 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28099.log 2026-02-22T20:34:20.683 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131328.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131328.log.gz 2026-02-22T20:34:20.683 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106434.log 2026-02-22T20:34:20.684 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28099.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28099.log.gz 2026-02-22T20:34:20.684 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30522.log 2026-02-22T20:34:20.685 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106434.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106434.log.gz 2026-02-22T20:34:20.685 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34528.log 2026-02-22T20:34:20.685 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128110.log.gz 2026-02-22T20:34:20.686 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30522.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30522.log.gz 2026-02-22T20:34:20.686 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22577.log 2026-02-22T20:34:20.686 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79319.log 2026-02-22T20:34:20.687 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34528.log.gz 2026-02-22T20:34:20.687 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22577.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108530.log 2026-02-22T20:34:20.687 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.22577.log.gz 2026-02-22T20:34:20.687 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122389.log 2026-02-22T20:34:20.688 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.79319.log.gz 2026-02-22T20:34:20.688 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108530.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108530.log.gz 2026-02-22T20:34:20.688 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119359.log 2026-02-22T20:34:20.688 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40130.log 2026-02-22T20:34:20.689 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122389.log.gz 2026-02-22T20:34:20.689 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119359.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119359.log.gz 2026-02-22T20:34:20.689 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40281.log 2026-02-22T20:34:20.690 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126131.log 2026-02-22T20:34:20.690 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40130.log.gz 2026-02-22T20:34:20.690 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40281.log: 88.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100749.log 2026-02-22T20:34:20.691 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40281.log.gz 2026-02-22T20:34:20.691 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116738.log 2026-02-22T20:34:20.691 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126131.log.gz 2026-02-22T20:34:20.691 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129459.log 2026-02-22T20:34:20.691 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116738.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.100749.log.gz 2026-02-22T20:34:20.691 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120959.log 2026-02-22T20:34:20.692 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129459.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.129459.log.gz 2026-02-22T20:34:20.692 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87049.log 2026-02-22T20:34:20.693 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120959.log: 94.4% 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.3.20155.log.gz 2026-02-22T20:34:20.693 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.101048.log.gz 2026-02-22T20:34:20.693 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120959.log.gz 2026-02-22T20:34:20.693 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30580.log 2026-02-22T20:34:20.694 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45299.log 2026-02-22T20:34:20.694 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87049.log.gz 2026-02-22T20:34:20.694 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85248.log 2026-02-22T20:34:20.694 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37435.log 2026-02-22T20:34:20.694 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30580.log.gz 2026-02-22T20:34:20.695 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45299.log.gz 2026-02-22T20:34:20.695 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60645.log 2026-02-22T20:34:20.695 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.85127.log 2026-02-22T20:34:20.696 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37435.log.gz/var/log/ceph/cluster2-client.mirror.60645.log: 2026-02-22T20:34:20.696 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60645.log.gz 2026-02-22T20:34:20.696 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41579.log 2026-02-22T20:34:20.696 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.0.85127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88209.log 2026-02-22T20:34:20.697 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41579.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41579.log.gz 2026-02-22T20:34:20.697 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84717.log 2026-02-22T20:34:20.697 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88209.log: 84.9% -- replaced with /var/log/ceph/cluster2-client.mirror.88209.log.gz 2026-02-22T20:34:20.698 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71908.log 2026-02-22T20:34:20.698 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84717.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84717.log.gz 2026-02-22T20:34:20.699 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52606.log 2026-02-22T20:34:20.699 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71908.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71908.log.gz 2026-02-22T20:34:20.699 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107413.log 2026-02-22T20:34:20.700 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52606.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52606.log.gz 2026-02-22T20:34:20.700 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90690.log 2026-02-22T20:34:20.701 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.107413.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.85127.log.gz 2026-02-22T20:34:20.702 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66138.log 2026-02-22T20:34:20.702 INFO:teuthology.orchestra.run.trial066.stderr: 92.5%/var/log/ceph/cluster1-client.mirror.90690.log: -- replaced with /var/log/ceph/cluster2-client.mirror.107413.log.gz 2026-02-22T20:34:20.702 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90690.log.gzgzip 2026-02-22T20:34:20.702 INFO:teuthology.orchestra.run.trial066.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91602.log 2026-02-22T20:34:20.703 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50418.log 2026-02-22T20:34:20.703 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91602.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66138.log.gz 2026-02-22T20:34:20.703 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91602.log.gz 2026-02-22T20:34:20.703 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29499.log 2026-02-22T20:34:20.704 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26390.log 2026-02-22T20:34:20.704 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50418.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50418.log.gz 2026-02-22T20:34:20.704 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29499.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29499.log.gz 2026-02-22T20:34:20.706 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28192.log 2026-02-22T20:34:20.707 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124047.log 2026-02-22T20:34:20.707 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26390.log: /var/log/ceph/cluster2-client.mirror.28192.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26390.log.gz 2026-02-22T20:34:20.707 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28192.log.gz 2026-02-22T20:34:20.707 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75543.log 2026-02-22T20:34:20.708 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44090.log 2026-02-22T20:34:20.708 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124047.log.gz 2026-02-22T20:34:20.708 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.75543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100803.log 2026-02-22T20:34:20.708 INFO:teuthology.orchestra.run.trial066.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.admin.75543.log.gz 2026-02-22T20:34:20.709 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44090.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44090.log.gz 2026-02-22T20:34:20.715 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61489.log.gz 2026-02-22T20:34:20.715 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55245.log 2026-02-22T20:34:20.716 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121696.log 2026-02-22T20:34:20.716 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100803.log.gz 2026-02-22T20:34:20.716 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95209.log 2026-02-22T20:34:20.716 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55245.log.gz 2026-02-22T20:34:20.716 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52356.log 2026-02-22T20:34:20.718 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121696.log: /var/log/ceph/cluster2-client.mirror.95209.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.121696.log.gz 2026-02-22T20:34:20.730 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116519.log 2026-02-22T20:34:20.731 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44241.log 2026-02-22T20:34:20.731 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116519.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116519.log.gz 2026-02-22T20:34:20.732 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61362.log 2026-02-22T20:34:20.732 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44241.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44241.log.gz 2026-02-22T20:34:20.733 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89106.log 2026-02-22T20:34:20.737 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.61362.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61362.log.gz 2026-02-22T20:34:20.737 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101741.log 2026-02-22T20:34:20.737 INFO:teuthology.orchestra.run.trial066.stderr: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52356.log.gz 2026-02-22T20:34:20.738 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29650.log 2026-02-22T20:34:20.738 INFO:teuthology.orchestra.run.trial066.stderr: 84.7%/var/log/ceph/cluster1-client.mirror.101741.log: -- replaced with /var/log/ceph/cluster2-client.mirror.89106.log.gz 2026-02-22T20:34:20.738 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130543.log 2026-02-22T20:34:20.738 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.101741.log.gz 2026-02-22T20:34:20.739 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125771.log 2026-02-22T20:34:20.739 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.130543.log: 88.2% 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.29650.log.gz 2026-02-22T20:34:20.740 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116738.log.gz 2026-02-22T20:34:20.740 INFO:teuthology.orchestra.run.trial066.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.130543.log.gz 2026-02-22T20:34:20.740 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92746.log 2026-02-22T20:34:20.741 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19705.log 2026-02-22T20:34:20.741 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125771.log: /var/log/ceph/cluster2-client.mirror.92746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26855.log 2026-02-22T20:34:20.741 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125771.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92746.log.gz 2026-02-22T20:34:20.741 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.741 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19705.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19705.log.gz 2026-02-22T20:34:20.742 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61816.log 2026-02-22T20:34:20.742 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94599.log 2026-02-22T20:34:20.742 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26855.log.gz 2026-02-22T20:34:20.742 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69583.log 2026-02-22T20:34:20.743 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61816.log.gz 2026-02-22T20:34:20.743 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124564.log 2026-02-22T20:34:20.743 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.94599.log: /var/log/ceph/cluster2-client.mirror.69583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93207.log 2026-02-22T20:34:20.743 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69583.log.gz 2026-02-22T20:34:20.743 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124564.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94599.log.gz 2026-02-22T20:34:20.743 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79634.log 2026-02-22T20:34:20.744 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93207.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93207.log.gz 2026-02-22T20:34:20.744 INFO:teuthology.orchestra.run.trial066.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124564.log.gz 2026-02-22T20:34:20.744 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130031.log 2026-02-22T20:34:20.745 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58820.log 2026-02-22T20:34:20.745 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79634.log: /var/log/ceph/cluster2-client.mirror.130031.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130031.log.gz 2026-02-22T20:34:20.745 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79634.log.gz 2026-02-22T20:34:20.746 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100838.log 2026-02-22T20:34:20.746 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48837.log 2026-02-22T20:34:20.746 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58820.log.gz 2026-02-22T20:34:20.746 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42881.log 2026-02-22T20:34:20.746 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.100838.log.gz 2026-02-22T20:34:20.746 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54594.log 2026-02-22T20:34:20.747 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48837.log: /var/log/ceph/cluster2-client.mirror.42881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111809.log 2026-02-22T20:34:20.747 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54594.log: 88.6% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48837.log.gz 2026-02-22T20:34:20.747 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42881.log.gz 2026-02-22T20:34:20.747 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54594.log.gz 2026-02-22T20:34:20.747 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125045.log 2026-02-22T20:34:20.747 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62095.log 2026-02-22T20:34:20.748 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.111809.log: /var/log/ceph/cluster2-client.mirror.125045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53199.log 2026-02-22T20:34:20.748 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62095.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125045.log.gz 2026-02-22T20:34:20.748 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62095.log.gz 2026-02-22T20:34:20.748 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47217.log 2026-02-22T20:34:20.749 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113634.log 2026-02-22T20:34:20.749 INFO:teuthology.orchestra.run.trial066.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111809.log.gz 2026-02-22T20:34:20.749 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53199.log.gz 2026-02-22T20:34:20.749 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47217.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47217.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31982.log 2026-02-22T20:34:20.749 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.749 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57610.log 2026-02-22T20:34:20.750 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113634.log: /var/log/ceph/cluster2-client.mirror.31982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56439.log 2026-02-22T20:34:20.750 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57610.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31982.log.gz 2026-02-22T20:34:20.750 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57610.log.gz 2026-02-22T20:34:20.750 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71536.log 2026-02-22T20:34:20.750 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.56439.log: 76.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56439.log.gz 2026-02-22T20:34:20.751 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65266.log 2026-02-22T20:34:20.751 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108778.log 2026-02-22T20:34:20.752 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71536.log.gz/var/log/ceph/cluster2-client.mirror.65266.log: 2026-02-22T20:34:20.752 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65266.log.gz 2026-02-22T20:34:20.752 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122651.log 2026-02-22T20:34:20.752 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108778.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108778.log.gz 2026-02-22T20:34:20.753 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87596.log 2026-02-22T20:34:20.753 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49860.log 2026-02-22T20:34:20.753 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87596.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.122651.log.gz 2026-02-22T20:34:20.754 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101462.log 2026-02-22T20:34:20.754 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49860.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49860.log.gz 2026-02-22T20:34:20.755 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43997.log 2026-02-22T20:34:20.755 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101462.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.101462.log.gz 2026-02-22T20:34:20.755 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19579.log 2026-02-22T20:34:20.756 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43997.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43997.log.gz 2026-02-22T20:34:20.756 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33970.log 2026-02-22T20:34:20.756 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19579.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19579.log.gz 2026-02-22T20:34:20.757 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116604.log 2026-02-22T20:34:20.757 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33970.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33970.log.gz 2026-02-22T20:34:20.757 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82055.log 2026-02-22T20:34:20.758 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116604.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116604.log.gz 2026-02-22T20:34:20.758 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110523.log 2026-02-22T20:34:20.759 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82055.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82055.log.gz 2026-02-22T20:34:20.759 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125996.log 2026-02-22T20:34:20.759 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110523.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110523.log.gz 2026-02-22T20:34:20.760 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56831.log 2026-02-22T20:34:20.760 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125996.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125996.log.gz 2026-02-22T20:34:20.761 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54873.log 2026-02-22T20:34:20.761 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56831.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56831.log.gz 2026-02-22T20:34:20.761 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62826.log 2026-02-22T20:34:20.762 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54873.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54873.log.gz 2026-02-22T20:34:20.762 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121117.log 2026-02-22T20:34:20.763 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62826.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62826.log.gz 2026-02-22T20:34:20.763 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69827.log 2026-02-22T20:34:20.764 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.121117.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121117.log.gz 2026-02-22T20:34:20.765 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74756.log 2026-02-22T20:34:20.765 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69827.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69827.log.gz 2026-02-22T20:34:20.765 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113024.log 2026-02-22T20:34:20.766 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74756.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74756.log.gz 2026-02-22T20:34:20.766 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23693.log 2026-02-22T20:34:20.767 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113024.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113024.log.gz 2026-02-22T20:34:20.767 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129925.log 2026-02-22T20:34:20.767 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23693.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.23693.log.gz 2026-02-22T20:34:20.768 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37679.log 2026-02-22T20:34:20.768 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129925.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.129925.log.gz 2026-02-22T20:34:20.768 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129632.log 2026-02-22T20:34:20.769 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37679.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37679.log.gz 2026-02-22T20:34:20.769 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90156.log 2026-02-22T20:34:20.776 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129632.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.89830.log.gz 2026-02-22T20:34:20.776 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121554.log 2026-02-22T20:34:20.777 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90156.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90156.log.gz 2026-02-22T20:34:20.777 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63778.log 2026-02-22T20:34:20.779 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121554.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121554.log.gz 2026-02-22T20:34:20.779 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30862.log 2026-02-22T20:34:20.780 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63778.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63778.log.gz 2026-02-22T20:34:20.780 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29464.log 2026-02-22T20:34:20.780 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30862.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30862.log.gz 2026-02-22T20:34:20.781 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58913.log 2026-02-22T20:34:20.781 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29464.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29464.log.gz 2026-02-22T20:34:20.782 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27692.log 2026-02-22T20:34:20.782 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58913.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58913.log.gz 2026-02-22T20:34:20.782 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35700.log 2026-02-22T20:34:20.783 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27692.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27692.log.gz 2026-02-22T20:34:20.783 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59378.log 2026-02-22T20:34:20.798 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35700.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97792.log.gz 2026-02-22T20:34:20.799 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36275.log 2026-02-22T20:34:20.799 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59378.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59378.log.gz 2026-02-22T20:34:20.800 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126154.log 2026-02-22T20:34:20.800 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36275.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36275.log.gz 2026-02-22T20:34:20.801 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.86308.log 2026-02-22T20:34:20.801 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126154.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126154.log.gz 2026-02-22T20:34:20.801 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106073.log 2026-02-22T20:34:20.818 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.0.86308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16474.log 2026-02-22T20:34:20.819 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106073.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106073.log.gz 2026-02-22T20:34:20.820 INFO:teuthology.orchestra.run.trial066.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.86308.log.gz 2026-02-22T20:34:20.821 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33098.log 2026-02-22T20:34:20.821 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.16474.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16474.log.gz 2026-02-22T20:34:20.821 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78829.log 2026-02-22T20:34:20.822 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33098.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33098.log.gz 2026-02-22T20:34:20.822 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42230.log 2026-02-22T20:34:20.823 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78829.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78829.log.gz 2026-02-22T20:34:20.823 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28720.log 2026-02-22T20:34:20.823 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42230.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42230.log.gz 2026-02-22T20:34:20.824 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29127.log 2026-02-22T20:34:20.824 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28720.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28720.log.gz 2026-02-22T20:34:20.825 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124025.log 2026-02-22T20:34:20.825 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29127.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29127.log.gz 2026-02-22T20:34:20.826 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113908.log 2026-02-22T20:34:20.827 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124025.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124025.log.gz 2026-02-22T20:34:20.827 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106329.log 2026-02-22T20:34:20.828 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113908.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113908.log.gz 2026-02-22T20:34:20.828 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83617.log 2026-02-22T20:34:20.829 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106329.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106329.log.gz 2026-02-22T20:34:20.829 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43125.log 2026-02-22T20:34:20.830 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83617.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83617.log.gz 2026-02-22T20:34:20.830 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116041.log 2026-02-22T20:34:20.831 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43125.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43125.log.gz 2026-02-22T20:34:20.831 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131522.log 2026-02-22T20:34:20.832 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116041.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116041.log.gz 2026-02-22T20:34:20.833 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45729.log 2026-02-22T20:34:20.834 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.131522.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.131522.log.gz 2026-02-22T20:34:20.834 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91701.log.gz 2026-02-22T20:34:20.834 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50383.log 2026-02-22T20:34:20.836 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77219.log 2026-02-22T20:34:20.836 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45729.log.gz 2026-02-22T20:34:20.836 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50383.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50383.log.gz 2026-02-22T20:34:20.836 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117445.log 2026-02-22T20:34:20.836 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127776.log 2026-02-22T20:34:20.836 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117445.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77219.log.gz 2026-02-22T20:34:20.836 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48372.log 2026-02-22T20:34:20.836 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127776.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127776.log.gz 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87455.log.gz 2026-02-22T20:34:20.837 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.837 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103696.log 2026-02-22T20:34:20.837 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48372.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48372.log.gz 2026-02-22T20:34:20.837 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53443.log 2026-02-22T20:34:20.838 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103696.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103696.log.gz 2026-02-22T20:34:20.838 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127166.log 2026-02-22T20:34:20.838 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53443.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53443.log.gz 2026-02-22T20:34:20.839 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60308.log 2026-02-22T20:34:20.839 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127189.log 2026-02-22T20:34:20.839 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127166.log: /var/log/ceph/cluster2-client.mirror.60308.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127166.log.gz 2026-02-22T20:34:20.840 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60308.log.gz 2026-02-22T20:34:20.840 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27913.log 2026-02-22T20:34:20.840 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102153.log 2026-02-22T20:34:20.841 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127189.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127189.log.gz 2026-02-22T20:34:20.841 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27913.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.27913.log.gz 2026-02-22T20:34:20.850 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93103.log.gz 2026-02-22T20:34:20.851 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109068.log 2026-02-22T20:34:20.851 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85886.log 2026-02-22T20:34:20.851 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.109068.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.102153.log.gz 2026-02-22T20:34:20.851 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20264.log 2026-02-22T20:34:20.852 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68897.log 2026-02-22T20:34:20.852 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85886.log: /var/log/ceph/cluster1-client.admin.20264.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20264.log.gz 2026-02-22T20:34:20.852 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- 92.6% /var/log/ceph/cluster2-client.mirror.19982.log 2026-02-22T20:34:20.852 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.109068.log.gz 2026-02-22T20:34:20.853 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68897.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68897.log.gz 2026-02-22T20:34:20.853 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72803.log 2026-02-22T20:34:20.853 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.19982.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79109.log 2026-02-22T20:34:20.854 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74140.log 2026-02-22T20:34:20.854 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72803.log.gz 2026-02-22T20:34:20.854 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80489.log 2026-02-22T20:34:20.855 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74140.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74140.log.gz 2026-02-22T20:34:20.856 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79109.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79109.log.gz 2026-02-22T20:34:20.857 INFO:teuthology.orchestra.run.trial066.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.19982.log.gz 2026-02-22T20:34:20.858 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104080.log 2026-02-22T20:34:20.858 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100099.log 2026-02-22T20:34:20.860 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.104080.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104080.log.gz 2026-02-22T20:34:20.860 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80489.log.gz 2026-02-22T20:34:20.861 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91388.log 2026-02-22T20:34:20.861 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88766.log 2026-02-22T20:34:20.863 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100099.log.gz/var/log/ceph/cluster1-client.mirror.91388.log: 2026-02-22T20:34:20.863 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.113634.log.gz 2026-02-22T20:34:20.863 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91388.log.gz 2026-02-22T20:34:20.863 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125141.log 2026-02-22T20:34:20.863 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58634.log 2026-02-22T20:34:20.863 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88766.log.gz/var/log/ceph/cluster1-client.mirror.125141.log: 2026-02-22T20:34:20.863 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78899.log 2026-02-22T20:34:20.863 INFO:teuthology.orchestra.run.trial066.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125141.log.gz 2026-02-22T20:34:20.864 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108991.log 2026-02-22T20:34:20.864 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58634.log.gz 2026-02-22T20:34:20.864 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99641.log 2026-02-22T20:34:20.865 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108991.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78899.log.gz 2026-02-22T20:34:20.865 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131233.log 2026-02-22T20:34:20.865 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.99641.log: -- replaced with /var/log/ceph/cluster2-client.mirror.108991.log.gz 2026-02-22T20:34:20.865 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56045.log 2026-02-22T20:34:20.865 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.99641.log.gz 2026-02-22T20:34:20.865 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131233.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131233.log.gz 2026-02-22T20:34:20.866 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19497.log 2026-02-22T20:34:20.866 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36494.log 2026-02-22T20:34:20.867 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56045.log.gz 2026-02-22T20:34:20.867 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19497.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19497.log.gz 2026-02-22T20:34:20.867 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113873.log 2026-02-22T20:34:20.867 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36494.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22240.log 2026-02-22T20:34:20.867 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108353.log 2026-02-22T20:34:20.867 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22240.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113873.log.gz 2026-02-22T20:34:20.867 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.22240.log.gz 2026-02-22T20:34:20.870 INFO:teuthology.orchestra.run.trial066.stderr: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36494.log.gz 2026-02-22T20:34:20.870 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36009.log 2026-02-22T20:34:20.870 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.108353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37993.log 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79949.log 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37993.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36009.log.gz 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108353.log.gz 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.37993.log.gz 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27599.log 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131570.log 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27599.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79949.log.gz 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91679.log 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.27599.log.gz 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.131570.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.131570.log.gz 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114971.log 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.91679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48651.log 2026-02-22T20:34:20.873 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122821.log 2026-02-22T20:34:20.874 INFO:teuthology.orchestra.run.trial066.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.48651.log: -- replaced with /var/log/ceph/cluster2-client.mirror.114971.log.gz 2026-02-22T20:34:20.875 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48651.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.56439.log 2026-02-22T20:34:20.875 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.875 INFO:teuthology.orchestra.run.trial066.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91679.log.gz 2026-02-22T20:34:20.875 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122821.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122821.log.gz 2026-02-22T20:34:20.875 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24565.log 2026-02-22T20:34:20.875 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.0.56439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62223.log 2026-02-22T20:34:20.875 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106469.log 2026-02-22T20:34:20.875 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62223.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.24565.log.gz 2026-02-22T20:34:20.876 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94943.log 2026-02-22T20:34:20.876 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62223.log.gz 2026-02-22T20:34:20.876 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106469.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106469.log.gz 2026-02-22T20:34:20.876 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62868.log 2026-02-22T20:34:20.877 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70513.log 2026-02-22T20:34:20.877 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62868.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62868.log.gz 2026-02-22T20:34:20.877 INFO:teuthology.orchestra.run.trial066.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.56439.log.gz 2026-02-22T20:34:20.890 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120163.log 2026-02-22T20:34:20.891 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61299.log 2026-02-22T20:34:20.891 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120163.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70513.log.gz 2026-02-22T20:34:20.892 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120163.log.gz 2026-02-22T20:34:20.892 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93420.log 2026-02-22T20:34:20.893 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105903.log 2026-02-22T20:34:20.893 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93420.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93420.log.gz 2026-02-22T20:34:20.893 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64150.log 2026-02-22T20:34:20.894 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105903.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105903.log.gz 2026-02-22T20:34:20.894 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32726.log 2026-02-22T20:34:20.895 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64150.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64150.log.gz 2026-02-22T20:34:20.896 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35424.log 2026-02-22T20:34:20.896 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32726.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61299.log.gz 2026-02-22T20:34:20.896 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32726.log.gz 2026-02-22T20:34:20.897 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57668.log 2026-02-22T20:34:20.897 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61110.log 2026-02-22T20:34:20.897 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.57668.log: -- replaced with /var/log/ceph/cluster2-client.mirror.35424.log.gz 2026-02-22T20:34:20.897 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57668.log.gz 2026-02-22T20:34:20.898 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64801.log 2026-02-22T20:34:20.898 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61110.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78304.log 2026-02-22T20:34:20.902 INFO:teuthology.orchestra.run.trial066.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.64801.log: -- replaced with /var/log/ceph/cluster2-client.mirror.61110.log.gz 2026-02-22T20:34:20.902 INFO:teuthology.orchestra.run.trial066.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2026-02-22T20:34:20.902 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64801.log.gz 2026-02-22T20:34:20.902 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111454.log 2026-02-22T20:34:20.902 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20425.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78304.log.gz 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80073.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster1-client.mirror.20425.log: -- replaced with /var/log/ceph/cluster2-client.mirror.111454.log.gz 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36931.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr: 58.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20425.log.gz 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24065.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80073.log.gz 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55675.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36931.log.gz 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38144.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.24065.log.gz/var/log/ceph/cluster2-client.mirror.55675.log: 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98806.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38144.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55675.log.gz 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92079.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38144.log.gz 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119416.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.92079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108813.log 2026-02-22T20:34:20.903 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119416.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92079.log.gz 2026-02-22T20:34:20.904 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71257.log 2026-02-22T20:34:20.904 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108813.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108813.log.gz 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119416.log.gz 2026-02-22T20:34:20.905 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:20.905 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56322.log 2026-02-22T20:34:20.905 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115329.log 2026-02-22T20:34:20.905 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71257.log.gz 2026-02-22T20:34:20.910 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19679.log 2026-02-22T20:34:20.911 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115329.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115329.log.gz 2026-02-22T20:34:20.922 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129275.log 2026-02-22T20:34:20.923 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19679.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19679.log.gz 2026-02-22T20:34:20.947 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111066.log 2026-02-22T20:34:20.947 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129275.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129275.log.gz 2026-02-22T20:34:20.947 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65673.log 2026-02-22T20:34:20.948 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111066.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111066.log.gz 2026-02-22T20:34:20.949 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38365.log 2026-02-22T20:34:20.949 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65673.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65673.log.gz 2026-02-22T20:34:20.950 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57389.log 2026-02-22T20:34:20.951 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38365.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38365.log.gz 2026-02-22T20:34:20.951 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59436.log 2026-02-22T20:34:20.951 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57389.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57389.log.gz 2026-02-22T20:34:20.952 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101674.log 2026-02-22T20:34:20.952 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59436.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59436.log.gz 2026-02-22T20:34:20.953 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45171.log 2026-02-22T20:34:20.953 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101674.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.101674.log.gz 2026-02-22T20:34:20.954 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59006.log 2026-02-22T20:34:20.954 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45171.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45171.log.gz 2026-02-22T20:34:20.955 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99793.log 2026-02-22T20:34:20.956 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59006.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84934.log.gz 2026-02-22T20:34:20.956 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59006.log.gz 2026-02-22T20:34:20.956 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118114.log 2026-02-22T20:34:20.956 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20476.log 2026-02-22T20:34:20.959 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99793.log.gz 2026-02-22T20:34:20.959 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118114.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118114.log.gz 2026-02-22T20:34:20.959 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64557.log 2026-02-22T20:34:20.959 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78724.log 2026-02-22T20:34:20.959 INFO:teuthology.orchestra.run.trial066.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.20476.log.gz 2026-02-22T20:34:20.959 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101766.log 2026-02-22T20:34:20.959 INFO:teuthology.orchestra.run.trial066.stderr: 89.0%/var/log/ceph/cluster2-client.mirror.78724.log: -- replaced with /var/log/ceph/cluster2-client.mirror.64557.log.gz 2026-02-22T20:34:20.959 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78724.log.gz 2026-02-22T20:34:20.960 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124422.log 2026-02-22T20:34:20.960 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85457.log 2026-02-22T20:34:20.961 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101766.log: /var/log/ceph/cluster2-client.mirror.124422.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101766.log.gz 2026-02-22T20:34:20.961 INFO:teuthology.orchestra.run.trial066.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124422.log.gz 2026-02-22T20:34:20.961 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62130.log 2026-02-22T20:34:20.961 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37117.log 2026-02-22T20:34:20.962 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85457.log: /var/log/ceph/cluster2-client.mirror.62130.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85457.log.gz 2026-02-22T20:34:20.962 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62130.log.gz 2026-02-22T20:34:20.962 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115294.log 2026-02-22T20:34:20.962 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37117.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27820.log 2026-02-22T20:34:20.963 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37117.log.gz 2026-02-22T20:34:20.965 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60587.log 2026-02-22T20:34:20.965 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103308.log 2026-02-22T20:34:20.965 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60587.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115294.log.gz 2026-02-22T20:34:20.965 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27820.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60587.log.gz 2026-02-22T20:34:20.965 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27820.log.gz 2026-02-22T20:34:20.965 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109695.log 2026-02-22T20:34:20.965 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103308.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115387.log 2026-02-22T20:34:20.965 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.103308.log.gz 2026-02-22T20:34:20.965 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.109695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50976.log 2026-02-22T20:34:20.967 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115387.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115387.log.gz 2026-02-22T20:34:20.967 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111330.log 2026-02-22T20:34:20.967 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50976.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50976.log.gz 2026-02-22T20:34:20.967 INFO:teuthology.orchestra.run.trial066.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109695.log.gz 2026-02-22T20:34:20.967 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73733.log 2026-02-22T20:34:20.967 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31552.log 2026-02-22T20:34:20.968 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111330.log.gz 2026-02-22T20:34:20.968 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105457.log 2026-02-22T20:34:20.968 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31552.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73733.log.gz 2026-02-22T20:34:20.968 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31552.log.gz 2026-02-22T20:34:20.968 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35235.log 2026-02-22T20:34:20.970 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47031.log 2026-02-22T20:34:20.970 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.105457.log: /var/log/ceph/cluster2-client.mirror.35235.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87596.log.gz 2026-02-22T20:34:20.970 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35235.log.gz 2026-02-22T20:34:20.970 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105457.log.gz 2026-02-22T20:34:20.971 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106023.log 2026-02-22T20:34:20.971 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115181.log 2026-02-22T20:34:20.971 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47031.log: /var/log/ceph/cluster2-client.mirror.106023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86624.log 2026-02-22T20:34:20.971 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47031.log.gz 2026-02-22T20:34:20.971 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106023.log.gz 2026-02-22T20:34:20.972 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.115181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95885.log 2026-02-22T20:34:20.973 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86624.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115181.log.gz 2026-02-22T20:34:20.973 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69362.log 2026-02-22T20:34:20.973 INFO:teuthology.orchestra.run.trial066.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86624.log.gz 2026-02-22T20:34:20.973 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.94942.log 2026-02-22T20:34:20.973 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.95885.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95885.log.gz 2026-02-22T20:34:20.973 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2026-02-22T20:34:20.973 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69362.log: /var/log/ceph/cluster1-client.mirror.1.94942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20340.log 2026-02-22T20:34:20.973 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2.audit.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69362.log.gz 2026-02-22T20:34:20.975 INFO:teuthology.orchestra.run.trial066.stderr: 92.9% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2026-02-22T20:34:20.987 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86230.log 2026-02-22T20:34:20.987 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82946.log 2026-02-22T20:34:20.987 INFO:teuthology.orchestra.run.trial066.stderr: 55.8% -- replaced with /var/log/ceph/cluster1-client.mirror.20340.log.gz 2026-02-22T20:34:20.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123183.log 2026-02-22T20:34:20.991 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82946.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82946.log.gz 2026-02-22T20:34:20.991 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52827.log 2026-02-22T20:34:20.993 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123183.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123183.log.gz 2026-02-22T20:34:20.993 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124189.log 2026-02-22T20:34:20.994 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52827.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52827.log.gz 2026-02-22T20:34:20.994 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84530.log 2026-02-22T20:34:20.994 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124189.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124189.log.gz 2026-02-22T20:34:20.995 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65452.log 2026-02-22T20:34:20.995 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84530.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84530.log.gz 2026-02-22T20:34:20.996 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130318.log 2026-02-22T20:34:20.996 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65452.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65452.log.gz 2026-02-22T20:34:20.996 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38851.log 2026-02-22T20:34:20.997 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130318.log: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130318.log.gz 2026-02-22T20:34:20.997 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44369.log 2026-02-22T20:34:20.997 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38851.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38851.log.gz 2026-02-22T20:34:20.998 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116131.log 2026-02-22T20:34:20.998 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44369.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44369.log.gz 2026-02-22T20:34:20.999 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37772.log 2026-02-22T20:34:21.006 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104799.log 2026-02-22T20:34:21.007 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37772.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37772.log.gz 2026-02-22T20:34:21.011 INFO:teuthology.orchestra.run.trial066.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.94942.log.gz 2026-02-22T20:34:21.014 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99010.log 2026-02-22T20:34:21.015 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104799.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104799.log.gz 2026-02-22T20:34:21.016 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72431.log 2026-02-22T20:34:21.016 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99010.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99010.log.gz 2026-02-22T20:34:21.017 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106192.log 2026-02-22T20:34:21.017 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72431.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72431.log.gz 2026-02-22T20:34:21.018 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74198.log 2026-02-22T20:34:21.018 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.106192.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106192.log.gz 2026-02-22T20:34:21.018 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131943.log 2026-02-22T20:34:21.019 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74198.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74198.log.gz 2026-02-22T20:34:21.019 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27041.log 2026-02-22T20:34:21.020 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131943.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131943.log.gz 2026-02-22T20:34:21.020 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127393.log 2026-02-22T20:34:21.021 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27041.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27041.log.gz 2026-02-22T20:34:21.021 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50232.log 2026-02-22T20:34:21.022 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127393.log: 55.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127393.log.gz 2026-02-22T20:34:21.022 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83575.log 2026-02-22T20:34:21.023 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50232.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50232.log.gz 2026-02-22T20:34:21.023 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80127.log 2026-02-22T20:34:21.024 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83575.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83575.log.gz 2026-02-22T20:34:21.024 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62316.log 2026-02-22T20:34:21.025 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80127.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80127.log.gz 2026-02-22T20:34:21.025 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83743.log 2026-02-22T20:34:21.026 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62316.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62316.log.gz 2026-02-22T20:34:21.026 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104520.log 2026-02-22T20:34:21.027 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83743.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83743.log.gz 2026-02-22T20:34:21.027 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65859.log 2026-02-22T20:34:21.028 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104520.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104520.log.gz 2026-02-22T20:34:21.028 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75959.log 2026-02-22T20:34:21.029 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65859.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65859.log.gz 2026-02-22T20:34:21.029 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59285.log 2026-02-22T20:34:21.030 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75959.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75959.log.gz 2026-02-22T20:34:21.030 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24472.log 2026-02-22T20:34:21.031 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59285.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59285.log.gz 2026-02-22T20:34:21.031 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28534.log 2026-02-22T20:34:21.032 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24472.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.24472.log.gz 2026-02-22T20:34:21.032 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131501.log 2026-02-22T20:34:21.033 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28534.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.28534.log.gz 2026-02-22T20:34:21.033 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125749.log 2026-02-22T20:34:21.034 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131501.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.131501.log.gz 2026-02-22T20:34:21.034 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76379.log 2026-02-22T20:34:21.035 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125749.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125749.log.gz 2026-02-22T20:34:21.035 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82326.log 2026-02-22T20:34:21.036 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76379.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76379.log.gz 2026-02-22T20:34:21.036 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103874.log 2026-02-22T20:34:21.036 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.82326.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82326.log.gz 2026-02-22T20:34:21.037 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42288.log 2026-02-22T20:34:21.037 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103874.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103874.log.gz 2026-02-22T20:34:21.038 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111645.log 2026-02-22T20:34:21.038 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42288.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42288.log.gz 2026-02-22T20:34:21.038 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121654.log 2026-02-22T20:34:21.039 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111645.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111645.log.gz 2026-02-22T20:34:21.039 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58413.log 2026-02-22T20:34:21.040 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121654.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121654.log.gz 2026-02-22T20:34:21.041 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35084.log 2026-02-22T20:34:21.041 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58413.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58413.log.gz 2026-02-22T20:34:21.041 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113612.log 2026-02-22T20:34:21.042 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35084.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35084.log.gz 2026-02-22T20:34:21.042 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111009.log 2026-02-22T20:34:21.044 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.113612.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113612.log.gz 2026-02-22T20:34:21.044 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69734.log 2026-02-22T20:34:21.045 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111009.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111009.log.gz 2026-02-22T20:34:21.045 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122693.log 2026-02-22T20:34:21.046 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69734.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69734.log.gz 2026-02-22T20:34:21.046 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80678.log 2026-02-22T20:34:21.048 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122693.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.122693.log.gz 2026-02-22T20:34:21.048 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22984.log 2026-02-22T20:34:21.049 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.80678.log: 84.4% -- replaced with /var/log/ceph/cluster1-client.mirror.80678.log.gz 2026-02-22T20:34:21.049 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26332.log 2026-02-22T20:34:21.049 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22984.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.22984.log.gz 2026-02-22T20:34:21.050 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104570.log 2026-02-22T20:34:21.050 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26332.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26332.log.gz 2026-02-22T20:34:21.051 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42416.log 2026-02-22T20:34:21.051 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104570.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104570.log.gz 2026-02-22T20:34:21.052 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30301.log 2026-02-22T20:34:21.052 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42416.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42416.log.gz 2026-02-22T20:34:21.052 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94766.log 2026-02-22T20:34:21.053 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30301.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30301.log.gz 2026-02-22T20:34:21.053 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40893.log 2026-02-22T20:34:21.054 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94766.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.94766.log.gz 2026-02-22T20:34:21.054 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40458.log 2026-02-22T20:34:21.055 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40893.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40893.log.gz 2026-02-22T20:34:21.055 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105818.log 2026-02-22T20:34:21.060 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40458.log: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40458.log.gz 2026-02-22T20:34:21.060 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119848.log 2026-02-22T20:34:21.061 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105818.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105818.log.gz 2026-02-22T20:34:21.061 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97693.log 2026-02-22T20:34:21.062 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119848.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119848.log.gz 2026-02-22T20:34:21.062 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84911.log 2026-02-22T20:34:21.063 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97693.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.97693.log.gz 2026-02-22T20:34:21.063 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26948.log 2026-02-22T20:34:21.063 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84911.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84911.log.gz 2026-02-22T20:34:21.064 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131401.log 2026-02-22T20:34:21.064 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26948.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26948.log.gz 2026-02-22T20:34:21.065 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109352.log 2026-02-22T20:34:21.065 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.131401.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131401.log.gz 2026-02-22T20:34:21.065 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60773.log 2026-02-22T20:34:21.066 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109352.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109352.log.gz 2026-02-22T20:34:21.066 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120309.log 2026-02-22T20:34:21.067 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60773.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60773.log.gz 2026-02-22T20:34:21.067 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52641.log 2026-02-22T20:34:21.069 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120309.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.120309.log.gz 2026-02-22T20:34:21.069 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37342.log 2026-02-22T20:34:21.070 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52641.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52641.log.gz 2026-02-22T20:34:21.070 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74663.log 2026-02-22T20:34:21.071 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37342.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37342.log.gz 2026-02-22T20:34:21.071 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76799.log 2026-02-22T20:34:21.072 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74663.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74663.log.gz 2026-02-22T20:34:21.072 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105853.log 2026-02-22T20:34:21.073 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76799.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76799.log.gz 2026-02-22T20:34:21.073 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90757.log 2026-02-22T20:34:21.073 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105853.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105853.log.gz 2026-02-22T20:34:21.074 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131263.log 2026-02-22T20:34:21.082 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90757.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94943.log.gz 2026-02-22T20:34:21.082 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75410.log 2026-02-22T20:34:21.083 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.131263.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.131263.log.gz 2026-02-22T20:34:21.086 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90255.log 2026-02-22T20:34:21.090 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75410.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75410.log.gz 2026-02-22T20:34:21.090 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120438.log 2026-02-22T20:34:21.099 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107159.log 2026-02-22T20:34:21.099 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120438.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120438.log.gz 2026-02-22T20:34:21.114 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130359.log 2026-02-22T20:34:21.115 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.107159.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107159.log.gz 2026-02-22T20:34:21.123 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42939.log 2026-02-22T20:34:21.123 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130359.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130359.log.gz 2026-02-22T20:34:21.124 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95209.log.gz 2026-02-22T20:34:21.143 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128043.log 2026-02-22T20:34:21.144 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42939.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42939.log.gz 2026-02-22T20:34:21.144 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51685.log 2026-02-22T20:34:21.144 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128043.log: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.128043.log.gz 2026-02-22T20:34:21.145 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99894.log 2026-02-22T20:34:21.145 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51685.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51685.log.gz 2026-02-22T20:34:21.146 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63592.log 2026-02-22T20:34:21.146 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99894.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99894.log.gz 2026-02-22T20:34:21.146 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75889.log 2026-02-22T20:34:21.147 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63592.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63592.log.gz 2026-02-22T20:34:21.147 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65580.log 2026-02-22T20:34:21.148 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75889.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75889.log.gz 2026-02-22T20:34:21.149 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91659.log 2026-02-22T20:34:21.149 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65580.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65580.log.gz 2026-02-22T20:34:21.150 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116887.log 2026-02-22T20:34:21.151 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91659.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91659.log.gz 2026-02-22T20:34:21.151 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130510.log 2026-02-22T20:34:21.152 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116887.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116887.log.gz 2026-02-22T20:34:21.152 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25925.log 2026-02-22T20:34:21.153 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.130510.log: 95.7% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84740.log.gz 2026-02-22T20:34:21.153 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.130510.log.gz 2026-02-22T20:34:21.153 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129870.log 2026-02-22T20:34:21.156 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36687.log 2026-02-22T20:34:21.156 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.25925.log.gz 2026-02-22T20:34:21.156 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74477.log 2026-02-22T20:34:21.156 INFO:teuthology.orchestra.run.trial066.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129870.log.gz 2026-02-22T20:34:21.156 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106679.log 2026-02-22T20:34:21.156 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36687.log.gz 2026-02-22T20:34:21.156 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60401.log 2026-02-22T20:34:21.157 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.106679.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74477.log.gz 2026-02-22T20:34:21.157 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.106679.log.gz 2026-02-22T20:34:21.159 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48279.log 2026-02-22T20:34:21.159 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39723.log 2026-02-22T20:34:21.159 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60401.log: /var/log/ceph/cluster2-client.mirror.48279.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60401.log.gz 2026-02-22T20:34:21.159 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48279.log.gz 2026-02-22T20:34:21.159 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102550.log 2026-02-22T20:34:21.159 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103567.log 2026-02-22T20:34:21.159 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39723.log.gz 2026-02-22T20:34:21.159 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127234.log 2026-02-22T20:34:21.159 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103567.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102550.log.gz 2026-02-22T20:34:21.161 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82447.log 2026-02-22T20:34:21.161 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127234.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127234.log.gz 2026-02-22T20:34:21.161 INFO:teuthology.orchestra.run.trial066.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103567.log.gz 2026-02-22T20:34:21.161 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87307.log 2026-02-22T20:34:21.162 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36429.log 2026-02-22T20:34:21.162 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82447.log: /var/log/ceph/cluster2-client.mirror.87307.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.82447.log.gz 2026-02-22T20:34:21.162 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80512.log 2026-02-22T20:34:21.164 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.36429.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.89129.log.gz 2026-02-22T20:34:21.165 INFO:teuthology.orchestra.run.trial066.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36429.log.gz 2026-02-22T20:34:21.175 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75318.log 2026-02-22T20:34:21.175 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47682.log 2026-02-22T20:34:21.176 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75318.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75318.log.gz 2026-02-22T20:34:21.176 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124351.log 2026-02-22T20:34:21.177 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47682.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47682.log.gz 2026-02-22T20:34:21.177 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91138.log 2026-02-22T20:34:21.179 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124351.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124351.log.gz 2026-02-22T20:34:21.179 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53013.log 2026-02-22T20:34:21.180 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91138.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91138.log.gz 2026-02-22T20:34:21.180 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40037.log 2026-02-22T20:34:21.180 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53013.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53013.log.gz 2026-02-22T20:34:21.181 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75430.log 2026-02-22T20:34:21.182 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40037.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40037.log.gz 2026-02-22T20:34:21.182 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30057.log 2026-02-22T20:34:21.185 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.75430.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75430.log.gz 2026-02-22T20:34:21.186 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89213.log 2026-02-22T20:34:21.186 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30057.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30057.log.gz 2026-02-22T20:34:21.187 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109583.log 2026-02-22T20:34:21.187 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89213.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89213.log.gz 2026-02-22T20:34:21.188 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82837.log 2026-02-22T20:34:21.188 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109583.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109583.log.gz 2026-02-22T20:34:21.188 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88030.log 2026-02-22T20:34:21.192 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.82837.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.82837.log.gz 2026-02-22T20:34:21.193 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119391.log 2026-02-22T20:34:21.195 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.88030.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.88030.log.gz 2026-02-22T20:34:21.195 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58041.log 2026-02-22T20:34:21.196 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.119391.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.119391.log.gz 2026-02-22T20:34:21.196 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20630.log 2026-02-22T20:34:21.197 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58041.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58041.log.gz 2026-02-22T20:34:21.198 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124402.log 2026-02-22T20:34:21.198 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20630.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.20630.log.gz 2026-02-22T20:34:21.198 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120581.log 2026-02-22T20:34:21.199 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124402.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124402.log.gz 2026-02-22T20:34:21.199 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94644.log 2026-02-22T20:34:21.200 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120581.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120581.log.gz 2026-02-22T20:34:21.200 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121912.log 2026-02-22T20:34:21.202 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.94644.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94644.log.gz 2026-02-22T20:34:21.202 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97398.log 2026-02-22T20:34:21.203 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121912.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121912.log.gz 2026-02-22T20:34:21.204 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115464.log 2026-02-22T20:34:21.205 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97398.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.97398.log.gz 2026-02-22T20:34:21.205 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45264.log 2026-02-22T20:34:21.206 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115464.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115464.log.gz 2026-02-22T20:34:21.206 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67347.log 2026-02-22T20:34:21.209 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45264.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45264.log.gz 2026-02-22T20:34:21.209 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35700.log.gz 2026-02-22T20:34:21.209 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50569.log 2026-02-22T20:34:21.209 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46229.log 2026-02-22T20:34:21.209 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67347.log: /var/log/ceph/cluster2-client.mirror.50569.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67347.log.gz 2026-02-22T20:34:21.209 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50569.log.gz 2026-02-22T20:34:21.210 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113202.log 2026-02-22T20:34:21.210 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86148.log 2026-02-22T20:34:21.210 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46229.log: /var/log/ceph/cluster2-client.mirror.113202.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46229.log.gz 2026-02-22T20:34:21.210 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113202.log.gz 2026-02-22T20:34:21.210 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100282.log 2026-02-22T20:34:21.212 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86005.log 2026-02-22T20:34:21.213 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86148.log: /var/log/ceph/cluster2-client.mirror.100282.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86148.log.gz 2026-02-22T20:34:21.213 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.100282.log.gz 2026-02-22T20:34:21.213 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81260.log 2026-02-22T20:34:21.213 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42974.log 2026-02-22T20:34:21.213 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86005.log: /var/log/ceph/cluster2-client.mirror.81260.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81260.log.gz 2026-02-22T20:34:21.213 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70478.log 2026-02-22T20:34:21.215 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42974.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42974.log.gz 2026-02-22T20:34:21.215 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76694.log 2026-02-22T20:34:21.215 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70478.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70478.log.gz 2026-02-22T20:34:21.215 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117796.log 2026-02-22T20:34:21.216 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76694.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76694.log.gz 2026-02-22T20:34:21.216 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130716.log 2026-02-22T20:34:21.216 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117796.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117796.log.gz 2026-02-22T20:34:21.216 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65917.log 2026-02-22T20:34:21.218 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.130716.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130716.log.gz 2026-02-22T20:34:21.218 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67626.log 2026-02-22T20:34:21.218 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65917.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65917.log.gz 2026-02-22T20:34:21.218 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67126.log 2026-02-22T20:34:21.219 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67626.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67626.log.gz 2026-02-22T20:34:21.219 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47717.log 2026-02-22T20:34:21.219 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67126.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67126.log.gz 2026-02-22T20:34:21.221 INFO:teuthology.orchestra.run.trial066.stderr: 95.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99574.log 2026-02-22T20:34:21.221 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.86230.log.gz 2026-02-22T20:34:21.221 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81693.log 2026-02-22T20:34:21.222 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47717.log.gz 2026-02-22T20:34:21.222 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99574.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48558.log 2026-02-22T20:34:21.222 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99574.log.gz 2026-02-22T20:34:21.222 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.81693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123854.log 2026-02-22T20:34:21.222 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48558.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48558.log.gz 2026-02-22T20:34:21.222 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91239.log 2026-02-22T20:34:21.223 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123854.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123854.log.gz 2026-02-22T20:34:21.224 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55836.log 2026-02-22T20:34:21.224 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91239.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.91239.log.gz 2026-02-22T20:34:21.225 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.81693.log.gz 2026-02-22T20:34:21.225 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92326.log 2026-02-22T20:34:21.225 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88616.log 2026-02-22T20:34:21.226 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.92326.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.92326.log.gz 2026-02-22T20:34:21.234 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23786.log 2026-02-22T20:34:21.247 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122622.log 2026-02-22T20:34:21.247 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23786.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.23786.log.gz 2026-02-22T20:34:21.268 INFO:teuthology.orchestra.run.trial066.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87307.log.gz 2026-02-22T20:34:21.268 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23821.log 2026-02-22T20:34:21.269 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122622.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122622.log.gz 2026-02-22T20:34:21.270 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28471.log 2026-02-22T20:34:21.270 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23821.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.23821.log.gz 2026-02-22T20:34:21.271 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126694.log 2026-02-22T20:34:21.271 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28471.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.28471.log.gz 2026-02-22T20:34:21.279 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52025.log 2026-02-22T20:34:21.279 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85886.log.gz 2026-02-22T20:34:21.279 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126694.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126694.log.gz 2026-02-22T20:34:21.281 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.84384.log 2026-02-22T20:34:21.281 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57855.log 2026-02-22T20:34:21.281 INFO:teuthology.orchestra.run.trial066.stderr: 88.6%/var/log/ceph/cluster2-client.mirror.2.84384.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52025.log.gz 2026-02-22T20:34:21.281 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24716.log 2026-02-22T20:34:21.281 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57855.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57855.log.gz 2026-02-22T20:34:21.282 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85836.log 2026-02-22T20:34:21.282 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24716.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24716.log.gz 2026-02-22T20:34:21.282 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118011.log 2026-02-22T20:34:21.295 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44427.log 2026-02-22T20:34:21.295 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.118011.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118011.log.gz 2026-02-22T20:34:21.303 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99240.log 2026-02-22T20:34:21.303 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44427.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44427.log.gz 2026-02-22T20:34:21.305 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85248.log.gz 2026-02-22T20:34:21.306 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34956.log 2026-02-22T20:34:21.307 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99240.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99240.log.gz 2026-02-22T20:34:21.307 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107480.log 2026-02-22T20:34:21.308 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34956.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34956.log.gz 2026-02-22T20:34:21.308 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90233.log 2026-02-22T20:34:21.309 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.107480.log: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.107480.log.gz 2026-02-22T20:34:21.309 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112812.log 2026-02-22T20:34:21.310 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.90233.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90233.log.gz 2026-02-22T20:34:21.311 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122531.log 2026-02-22T20:34:21.311 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.112812.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112812.log.gz 2026-02-22T20:34:21.313 INFO:teuthology.orchestra.run.trial066.stderr:gzip 95.7% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.87186.log.gz -- 2026-02-22T20:34:21.314 INFO:teuthology.orchestra.run.trial066.stderr: /var/log/ceph/cluster2-client.mirror.56157.log 2026-02-22T20:34:21.314 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93339.log 2026-02-22T20:34:21.314 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122531.log.gz 2026-02-22T20:34:21.314 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56157.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56157.log.gz 2026-02-22T20:34:21.314 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76099.log 2026-02-22T20:34:21.314 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19471.log 2026-02-22T20:34:21.314 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93339.log.gz 2026-02-22T20:34:21.314 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76099.log: gzip -5 --verbose -- 89.1% /var/log/ceph/cluster2-client.mirror.72896.log -- replaced with /var/log/ceph/cluster2-client.mirror.76099.log.gz 2026-02-22T20:34:21.314 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123534.log 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19471.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19471.log.gz 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123905.log 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123534.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72896.log.gz 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129429.log 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.123534.log.gz 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103196.log 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129429.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123905.log.gz 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33226.log 2026-02-22T20:34:21.319 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103196.log: 88.3% 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103196.log.gz 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.129429.log.gz 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83922.log 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109717.log 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33226.log.gz/var/log/ceph/cluster1-client.mirror.83922.log: 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.83922.log.gz 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19420.log 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131841.log 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19420.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19420.log.gz 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51034.log 2026-02-22T20:34:21.320 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131841.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131841.log.gz 2026-02-22T20:34:21.322 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19653.log 2026-02-22T20:34:21.322 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51034.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51034.log.gz 2026-02-22T20:34:21.322 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130778.log 2026-02-22T20:34:21.322 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19653.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19653.log.gz 2026-02-22T20:34:21.323 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44276.log 2026-02-22T20:34:21.323 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130778.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.84384.log.gz 2026-02-22T20:34:21.323 INFO:teuthology.orchestra.run.trial066.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130778.log.gz 2026-02-22T20:34:21.326 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119654.log 2026-02-22T20:34:21.326 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93709.log 2026-02-22T20:34:21.326 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44276.log.gz 2026-02-22T20:34:21.326 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119654.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119654.log.gz 2026-02-22T20:34:21.326 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92249.log 2026-02-22T20:34:21.326 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129843.log 2026-02-22T20:34:21.327 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93709.log: /var/log/ceph/cluster2-client.mirror.92249.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93709.log.gz 2026-02-22T20:34:21.327 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92249.log.gz 2026-02-22T20:34:21.327 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131043.log 2026-02-22T20:34:21.327 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86262.log 2026-02-22T20:34:21.327 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129843.log: /var/log/ceph/cluster2-client.mirror.131043.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129843.log.gz 2026-02-22T20:34:21.327 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131043.log.gz 2026-02-22T20:34:21.327 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97363.log 2026-02-22T20:34:21.327 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38795.log 2026-02-22T20:34:21.327 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86262.log: /var/log/ceph/cluster2-client.mirror.97363.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86262.log.gz 2026-02-22T20:34:21.330 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.97363.log.gz 2026-02-22T20:34:21.330 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131426.log 2026-02-22T20:34:21.330 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38795.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38795.log.gz 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125951.log 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93776.log 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.131426.log: /var/log/ceph/cluster1-client.mirror.125951.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131426.log.gz 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125951.log.gz 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102786.log 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.93776.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.93776.log.gz 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67312.log 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91850.log 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.102786.log: /var/log/ceph/cluster2-client.mirror.67312.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117445.log.gz 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102786.log.gz 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67312.log.gz 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126671.log 2026-02-22T20:34:21.331 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22298.log 2026-02-22T20:34:21.334 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91850.log: /var/log/ceph/cluster1-client.mirror.126671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85127.log 2026-02-22T20:34:21.334 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126671.log.gz 2026-02-22T20:34:21.334 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91850.log.gz 2026-02-22T20:34:21.334 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128661.log 2026-02-22T20:34:21.334 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.22298.log.gz 2026-02-22T20:34:21.334 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92047.log 2026-02-22T20:34:21.334 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.85127.log: /var/log/ceph/cluster1-client.mirror.128661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80404.log 2026-02-22T20:34:21.334 INFO:teuthology.orchestra.run.trial066.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.128661.log.gz 2026-02-22T20:34:21.335 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92047.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.85127.log.gz 2026-02-22T20:34:21.335 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92047.log.gz 2026-02-22T20:34:21.335 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110831.log 2026-02-22T20:34:21.335 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81673.log 2026-02-22T20:34:21.335 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80404.log: /var/log/ceph/cluster2-client.mirror.110831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128084.log 2026-02-22T20:34:21.335 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110831.log.gz 2026-02-22T20:34:21.336 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81673.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80404.log.gz 2026-02-22T20:34:21.336 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116825.log 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66603.log 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81673.log.gz 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128084.log: /var/log/ceph/cluster2-client.mirror.116825.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.128084.log.gz 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.116825.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42195.log 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100474.log 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66603.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89541.log 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42195.log: /var/log/ceph/cluster1-client.mirror.100474.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42195.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43625.log 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:21.339 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.100474.log.gz 2026-02-22T20:34:21.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104263.log 2026-02-22T20:34:21.343 INFO:teuthology.orchestra.run.trial066.stderr: 3.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89541.log.gz 2026-02-22T20:34:21.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118601.log 2026-02-22T20:34:21.343 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43625.log.gz 2026-02-22T20:34:21.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83554.log 2026-02-22T20:34:21.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104263.log: /var/log/ceph/cluster2-client.mirror.118601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63113.log 2026-02-22T20:34:21.343 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104263.log.gz/var/log/ceph/cluster2-client.mirror.83554.log: 2026-02-22T20:34:21.343 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118601.log.gz 2026-02-22T20:34:21.343 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45822.log 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83554.log.gz 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29313.log 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.63113.log: /var/log/ceph/cluster2-client.mirror.45822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34621.log 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29313.log: 89.1% 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.63113.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.45822.log.gz 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29313.log.gz 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35177.log 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93801.log 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34621.log: /var/log/ceph/cluster2-client.mirror.35177.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59994.log 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98806.log.gz 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93801.log: -- replaced with /var/log/ceph/cluster2-client.mirror.34621.log.gz 2026-02-22T20:34:21.344 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35177.log.gz 2026-02-22T20:34:21.350 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61665.log 2026-02-22T20:34:21.350 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51592.log 2026-02-22T20:34:21.350 INFO:teuthology.orchestra.run.trial066.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.61665.log: -- replaced with /var/log/ceph/cluster2-client.mirror.59994.log.gz 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93801.log.gz 2026-02-22T20:34:21.350 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:21.350 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117599.log 2026-02-22T20:34:21.350 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61665.log.gz 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125434.log 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51592.log: /var/log/ceph/cluster2-client.mirror.117599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34400.log 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51592.log.gz 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.117599.log.gz 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24658.log 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125434.log.gz 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45357.log 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24658.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34400.log.gz 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106364.log 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45357.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.24658.log.gz 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115830.log 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45357.log.gz 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115541.log 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115830.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118880.log 2026-02-22T20:34:21.351 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106364.log.gz 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115830.log.gz/var/log/ceph/cluster2-client.mirror.115541.log: 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115541.log.gz 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91204.log 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37958.log 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36384.log 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118880.log.gz 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.37958.log: -- replaced with /var/log/ceph/cluster2-client.mirror.91204.log.gz 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114062.log 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37958.log.gz 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.36384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71966.log 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66510.log 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.36384.log.gz 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114062.log: /var/log/ceph/cluster2-client.mirror.71966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57296.log 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr: 89.0%/var/log/ceph/cluster2-client.mirror.66510.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114062.log.gz 2026-02-22T20:34:21.352 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.71966.log.gz 2026-02-22T20:34:21.353 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84355.log 2026-02-22T20:34:21.353 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66510.log.gz 2026-02-22T20:34:21.353 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57296.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57296.log.gz 2026-02-22T20:34:21.354 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29220.log 2026-02-22T20:34:21.354 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127144.log 2026-02-22T20:34:21.354 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22891.log 2026-02-22T20:34:21.354 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29220.log.gz/var/log/ceph/cluster1-client.mirror.127144.log: 2026-02-22T20:34:21.354 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131451.log 2026-02-22T20:34:21.354 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127144.log.gz 2026-02-22T20:34:21.354 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22891.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.22891.log.gz 2026-02-22T20:34:21.368 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71443.log 2026-02-22T20:34:21.368 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31331.log 2026-02-22T20:34:21.368 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131451.log: /var/log/ceph/cluster2-client.mirror.71443.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.131451.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.86005.log 2026-02-22T20:34:21.368 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:21.368 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71443.log.gz 2026-02-22T20:34:21.368 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80038.log 2026-02-22T20:34:21.369 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.0.86005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61017.log 2026-02-22T20:34:21.370 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31331.log.gz 2026-02-22T20:34:21.370 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44927.log 2026-02-22T20:34:21.370 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61017.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80038.log.gz 2026-02-22T20:34:21.370 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61017.log.gz 2026-02-22T20:34:21.370 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36352.log 2026-02-22T20:34:21.370 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107433.log 2026-02-22T20:34:21.371 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44927.log: /var/log/ceph/cluster2-client.mirror.36352.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44927.log.gz 2026-02-22T20:34:21.371 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36352.log.gz 2026-02-22T20:34:21.371 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128435.log 2026-02-22T20:34:21.371 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106259.log 2026-02-22T20:34:21.372 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.107433.log: /var/log/ceph/cluster2-client.mirror.128435.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128435.log.gz 2026-02-22T20:34:21.372 INFO:teuthology.orchestra.run.trial066.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.107433.log.gz 2026-02-22T20:34:21.372 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110885.log 2026-02-22T20:34:21.372 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78934.log 2026-02-22T20:34:21.373 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.106259.log: /var/log/ceph/cluster2-client.mirror.110885.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110885.log.gz 2026-02-22T20:34:21.379 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20055.log 2026-02-22T20:34:21.379 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78934.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78934.log.gz 2026-02-22T20:34:21.390 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36652.log 2026-02-22T20:34:21.398 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20055.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109717.log.gz 2026-02-22T20:34:21.414 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126739.log 2026-02-22T20:34:21.415 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36652.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36652.log.gz 2026-02-22T20:34:21.415 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67967.log 2026-02-22T20:34:21.415 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126739.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126739.log.gz 2026-02-22T20:34:21.416 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20306.log 2026-02-22T20:34:21.416 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67967.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67967.log.gz 2026-02-22T20:34:21.417 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72094.log 2026-02-22T20:34:21.417 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.20306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94395.log 2026-02-22T20:34:21.417 INFO:teuthology.orchestra.run.trial066.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.20306.log.gz 2026-02-22T20:34:21.418 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72094.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72094.log.gz 2026-02-22T20:34:21.418 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122367.log 2026-02-22T20:34:21.419 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94395.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94395.log.gz 2026-02-22T20:34:21.419 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122054.log 2026-02-22T20:34:21.420 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122367.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122367.log.gz 2026-02-22T20:34:21.420 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62935.log 2026-02-22T20:34:21.422 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122054.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122054.log.gz 2026-02-22T20:34:21.422 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102754.log 2026-02-22T20:34:21.423 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.62935.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62935.log.gz 2026-02-22T20:34:21.423 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81155.log 2026-02-22T20:34:21.424 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102754.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102754.log.gz 2026-02-22T20:34:21.424 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33133.log 2026-02-22T20:34:21.424 INFO:teuthology.orchestra.run.trial066.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.20055.log.gz 2026-02-22T20:34:21.425 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35758.log 2026-02-22T20:34:21.425 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33133.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81155.log.gz 2026-02-22T20:34:21.425 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33133.log.gz 2026-02-22T20:34:21.425 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77324.log 2026-02-22T20:34:21.426 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.35758.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.35758.log.gz 2026-02-22T20:34:21.426 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93036.log 2026-02-22T20:34:21.426 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78584.log 2026-02-22T20:34:21.427 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77324.log: /var/log/ceph/cluster1-client.mirror.93036.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77324.log.gz 2026-02-22T20:34:21.427 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95110.log 2026-02-22T20:34:21.427 INFO:teuthology.orchestra.run.trial066.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93036.log.gz 2026-02-22T20:34:21.428 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123783.log 2026-02-22T20:34:21.428 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78584.log.gz 2026-02-22T20:34:21.428 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.95110.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.95110.log.gz 2026-02-22T20:34:21.428 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90079.log 2026-02-22T20:34:21.429 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118332.log 2026-02-22T20:34:21.429 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123783.log: /var/log/ceph/cluster2-client.mirror.90079.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.90757.log.gz 2026-02-22T20:34:21.429 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.90079.log.gz 2026-02-22T20:34:21.429 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101959.log 2026-02-22T20:34:21.430 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.118332.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30897.log 2026-02-22T20:34:21.430 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101959.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123783.log.gz 2026-02-22T20:34:21.430 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.101959.log.gz 2026-02-22T20:34:21.430 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59564.log 2026-02-22T20:34:21.431 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77954.log 2026-02-22T20:34:21.431 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30897.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118332.log.gz 2026-02-22T20:34:21.431 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59564.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30897.log.gz 2026-02-22T20:34:21.431 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59564.log.gz 2026-02-22T20:34:21.431 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121605.log 2026-02-22T20:34:21.432 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57424.log 2026-02-22T20:34:21.432 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121605.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77954.log.gz 2026-02-22T20:34:21.432 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35270.log 2026-02-22T20:34:21.432 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121605.log.gz 2026-02-22T20:34:21.432 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36240.log 2026-02-22T20:34:21.432 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57424.log: /var/log/ceph/cluster2-client.mirror.35270.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57424.log.gz 2026-02-22T20:34:21.433 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35270.log.gz 2026-02-22T20:34:21.433 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103231.log 2026-02-22T20:34:21.433 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27355.log 2026-02-22T20:34:21.434 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36240.log: /var/log/ceph/cluster2-client.mirror.103231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36604.log 2026-02-22T20:34:21.434 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36240.log.gz 2026-02-22T20:34:21.434 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.103231.log.gz 2026-02-22T20:34:21.434 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24844.log 2026-02-22T20:34:21.434 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27355.log.gz 2026-02-22T20:34:21.434 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78164.log 2026-02-22T20:34:21.435 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36604.log: /var/log/ceph/cluster2-client.mirror.24844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107194.log 2026-02-22T20:34:21.435 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24844.log.gz/var/log/ceph/cluster2-client.mirror.78164.log: 2026-02-22T20:34:21.435 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78164.log.gz 2026-02-22T20:34:21.436 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84553.log 2026-02-22T20:34:21.436 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.107194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102897.log 2026-02-22T20:34:21.436 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107194.log.gz 2026-02-22T20:34:21.436 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48058.log 2026-02-22T20:34:21.437 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102897.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.102897.log.gz 2026-02-22T20:34:21.437 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116716.log 2026-02-22T20:34:21.438 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48058.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48058.log.gz 2026-02-22T20:34:21.438 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106237.log 2026-02-22T20:34:21.439 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.116716.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116716.log.gz 2026-02-22T20:34:21.440 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100714.log 2026-02-22T20:34:21.441 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.106237.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106237.log.gz 2026-02-22T20:34:21.442 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130297.log 2026-02-22T20:34:21.442 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100714.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100714.log.gz 2026-02-22T20:34:21.442 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43683.log 2026-02-22T20:34:21.443 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116131.log.gz 2026-02-22T20:34:21.444 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130297.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130297.log.gz 2026-02-22T20:34:21.444 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120924.log 2026-02-22T20:34:21.445 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96804.log 2026-02-22T20:34:21.445 INFO:teuthology.orchestra.run.trial066.stderr: 88.6%/var/log/ceph/cluster2-client.mirror.120924.log: -- replaced with /var/log/ceph/cluster2-client.mirror.43683.log.gz 2026-02-22T20:34:21.445 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120924.log.gz 2026-02-22T20:34:21.445 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128495.log 2026-02-22T20:34:21.446 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.96804.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96804.log.gz 2026-02-22T20:34:21.446 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119106.log 2026-02-22T20:34:21.447 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23600.log 2026-02-22T20:34:21.447 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119106.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128495.log.gz 2026-02-22T20:34:21.447 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119106.log.gz 2026-02-22T20:34:21.447 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103273.log 2026-02-22T20:34:21.448 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105037.log 2026-02-22T20:34:21.448 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.23600.log.gz 2026-02-22T20:34:21.448 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103273.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103273.log.gz 2026-02-22T20:34:21.448 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98526.log 2026-02-22T20:34:21.448 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105037.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105037.log.gz 2026-02-22T20:34:21.449 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114763.log 2026-02-22T20:34:21.449 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92348.log 2026-02-22T20:34:21.449 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98526.log.gz 2026-02-22T20:34:21.450 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.114763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99275.log 2026-02-22T20:34:21.458 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98167.log 2026-02-22T20:34:21.459 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99275.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99275.log.gz 2026-02-22T20:34:21.469 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.114763.log.gz 2026-02-22T20:34:21.470 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69490.log 2026-02-22T20:34:21.471 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98167.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129632.log.gz 2026-02-22T20:34:21.471 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98167.log.gz 2026-02-22T20:34:21.474 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24937.log 2026-02-22T20:34:21.475 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20374.log 2026-02-22T20:34:21.475 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69490.log.gz 2026-02-22T20:34:21.476 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43032.log 2026-02-22T20:34:21.476 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.24937.log.gz 2026-02-22T20:34:21.476 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20374.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20374.log.gz 2026-02-22T20:34:21.477 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41172.log 2026-02-22T20:34:21.477 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43032.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131546.log 2026-02-22T20:34:21.477 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43032.log.gz 2026-02-22T20:34:21.478 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121767.log 2026-02-22T20:34:21.478 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131546.log: 88.6% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131546.log.gz 2026-02-22T20:34:21.478 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41172.log.gz 2026-02-22T20:34:21.478 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103758.log 2026-02-22T20:34:21.478 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118551.log 2026-02-22T20:34:21.479 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103758.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103758.log.gz 2026-02-22T20:34:21.479 INFO:teuthology.orchestra.run.trial066.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121767.log.gz 2026-02-22T20:34:21.480 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126941.log 2026-02-22T20:34:21.480 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115576.log 2026-02-22T20:34:21.481 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126941.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118551.log.gz 2026-02-22T20:34:21.481 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126941.log.gz 2026-02-22T20:34:21.481 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122196.log 2026-02-22T20:34:21.481 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85786.log 2026-02-22T20:34:21.482 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115576.log: /var/log/ceph/cluster2-client.mirror.122196.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115576.log.gz 2026-02-22T20:34:21.482 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98410.log 2026-02-22T20:34:21.483 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85786.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122196.log.gz 2026-02-22T20:34:21.483 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124635.log 2026-02-22T20:34:21.484 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98410.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.98410.log.gz 2026-02-22T20:34:21.484 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121676.log 2026-02-22T20:34:21.486 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124635.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124635.log.gz 2026-02-22T20:34:21.486 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.20081.log 2026-02-22T20:34:21.487 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121676.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.121676.log.gz 2026-02-22T20:34:21.487 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128834.log 2026-02-22T20:34:21.487 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.0.20081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.19982.log 2026-02-22T20:34:21.489 INFO:teuthology.orchestra.run.trial066.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.20081.log.gz 2026-02-22T20:34:21.502 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109387.log 2026-02-22T20:34:21.510 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.0.19982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125861.log 2026-02-22T20:34:21.511 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109387.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109387.log.gz 2026-02-22T20:34:21.519 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125164.log 2026-02-22T20:34:21.519 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125861.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125861.log.gz 2026-02-22T20:34:21.521 INFO:teuthology.orchestra.run.trial066.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55836.log.gz 2026-02-22T20:34:21.524 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92348.log.gz 2026-02-22T20:34:21.534 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29836.log 2026-02-22T20:34:21.535 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125164.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125164.log.gz 2026-02-22T20:34:21.536 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99045.log 2026-02-22T20:34:21.537 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29836.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29836.log.gz 2026-02-22T20:34:21.537 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70327.log 2026-02-22T20:34:21.537 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99045.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99045.log.gz 2026-02-22T20:34:21.538 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46322.log 2026-02-22T20:34:21.538 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70327.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70327.log.gz 2026-02-22T20:34:21.539 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113078.log 2026-02-22T20:34:21.539 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46322.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46322.log.gz 2026-02-22T20:34:21.539 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91413.log 2026-02-22T20:34:21.540 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113078.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113078.log.gz 2026-02-22T20:34:21.540 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51406.log 2026-02-22T20:34:21.542 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91413.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91413.log.gz 2026-02-22T20:34:21.542 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94430.log 2026-02-22T20:34:21.543 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51406.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51406.log.gz 2026-02-22T20:34:21.543 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71164.log 2026-02-22T20:34:21.544 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94430.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.94430.log.gz 2026-02-22T20:34:21.544 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78094.log 2026-02-22T20:34:21.545 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71164.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71164.log.gz 2026-02-22T20:34:21.545 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122879.log 2026-02-22T20:34:21.546 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78094.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78094.log.gz 2026-02-22T20:34:21.546 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111680.log 2026-02-22T20:34:21.546 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122879.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122879.log.gz 2026-02-22T20:34:21.547 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95796.log 2026-02-22T20:34:21.547 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111680.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111680.log.gz 2026-02-22T20:34:21.548 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122722.log 2026-02-22T20:34:21.549 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.95796.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.95796.log.gz 2026-02-22T20:34:21.549 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118820.log 2026-02-22T20:34:21.550 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122722.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122722.log.gz 2026-02-22T20:34:21.550 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118312.log 2026-02-22T20:34:21.554 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118820.log: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85836.log.gz 2026-02-22T20:34:21.555 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123141.log 2026-02-22T20:34:21.556 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118312.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118312.log.gz 2026-02-22T20:34:21.563 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.90255.log.gz 2026-02-22T20:34:21.571 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63045.log 2026-02-22T20:34:21.571 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89236.log 2026-02-22T20:34:21.572 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63045.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123141.log.gz 2026-02-22T20:34:21.572 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79879.log 2026-02-22T20:34:21.575 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89236.log: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63045.log.gz 2026-02-22T20:34:21.575 INFO:teuthology.orchestra.run.trial066.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.118820.log.gz 2026-02-22T20:34:21.575 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39200.log 2026-02-22T20:34:21.576 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57331.log 2026-02-22T20:34:21.576 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79879.log: /var/log/ceph/cluster2-client.mirror.39200.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39200.log.gz 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.79879.log.gz 2026-02-22T20:34:21.576 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:21.577 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104662.log 2026-02-22T20:34:21.577 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29871.log 2026-02-22T20:34:21.578 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57331.log: /var/log/ceph/cluster2-client.mirror.104662.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57331.log.gz 2026-02-22T20:34:21.578 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36198.log 2026-02-22T20:34:21.578 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29871.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.104662.log.gz 2026-02-22T20:34:21.579 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29871.log.gz 2026-02-22T20:34:21.579 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39665.log 2026-02-22T20:34:21.579 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112448.log 2026-02-22T20:34:21.579 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106259.log.gz 2026-02-22T20:34:21.580 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36198.log.gz 2026-02-22T20:34:21.580 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32447.log 2026-02-22T20:34:21.580 INFO:teuthology.orchestra.run.trial066.stderr: 88.5%/var/log/ceph/cluster1-client.mirror.112448.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39665.log.gz 2026-02-22T20:34:21.580 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116016.log 2026-02-22T20:34:21.581 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115025.log 2026-02-22T20:34:21.581 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32447.log.gz 2026-02-22T20:34:21.581 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.116016.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112448.log.gz 2026-02-22T20:34:21.581 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66417.log 2026-02-22T20:34:21.581 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116016.log.gz 2026-02-22T20:34:21.582 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72280.log 2026-02-22T20:34:21.582 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115025.log.gz 2026-02-22T20:34:21.582 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66417.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94532.log 2026-02-22T20:34:21.582 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66417.log.gz 2026-02-22T20:34:21.582 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125726.log 2026-02-22T20:34:21.583 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72280.log: /var/log/ceph/cluster2-client.mirror.94532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84872.log 2026-02-22T20:34:21.583 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72280.log.gz 2026-02-22T20:34:21.583 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.94532.log.gz/var/log/ceph/cluster1-client.mirror.125726.log: 2026-02-22T20:34:21.583 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125726.log.gz 2026-02-22T20:34:21.583 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68432.log 2026-02-22T20:34:21.583 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35578.log 2026-02-22T20:34:21.584 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84872.log: /var/log/ceph/cluster2-client.mirror.68432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44462.log 2026-02-22T20:34:21.584 INFO:teuthology.orchestra.run.trial066.stderr: 89.0%/var/log/ceph/cluster2-client.mirror.35578.log: -- replaced with /var/log/ceph/cluster2-client.mirror.68432.log.gz 2026-02-22T20:34:21.584 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35578.log.gz 2026-02-22T20:34:21.585 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97871.log 2026-02-22T20:34:21.585 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80512.log.gz 2026-02-22T20:34:21.585 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130568.log 2026-02-22T20:34:21.585 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44462.log.gz 2026-02-22T20:34:21.586 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97658.log 2026-02-22T20:34:21.586 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97871.log.gz 2026-02-22T20:34:21.586 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130568.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33656.log 2026-02-22T20:34:21.586 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.97658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76939.log 2026-02-22T20:34:21.587 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130568.log.gz 2026-02-22T20:34:21.587 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33656.log: -- replaced with /var/log/ceph/cluster2-client.mirror.97658.log.gz 2026-02-22T20:34:21.587 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76624.log 2026-02-22T20:34:21.587 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.33656.log.gz 2026-02-22T20:34:21.587 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123396.log 2026-02-22T20:34:21.588 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76939.log.gz 2026-02-22T20:34:21.588 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20698.log 2026-02-22T20:34:21.588 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76624.log.gz 2026-02-22T20:34:21.588 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125929.log 2026-02-22T20:34:21.588 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20698.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63172.log 2026-02-22T20:34:21.589 INFO:teuthology.orchestra.run.trial066.stderr: 64.7% -- replaced with /var/log/ceph/cluster2-client.mirror.20698.log.gz 2026-02-22T20:34:21.589 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125929.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125929.log.gz 2026-02-22T20:34:21.589 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85000.log 2026-02-22T20:34:21.589 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63172.log: 90.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58320.log 2026-02-22T20:34:21.590 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.123396.log.gz 2026-02-22T20:34:21.590 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38179.log 2026-02-22T20:34:21.590 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58320.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58320.log.gz 2026-02-22T20:34:21.591 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27320.log 2026-02-22T20:34:21.591 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38179.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38179.log.gz 2026-02-22T20:34:21.592 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30115.log 2026-02-22T20:34:21.592 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27320.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27320.log.gz 2026-02-22T20:34:21.593 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41916.log 2026-02-22T20:34:21.593 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30115.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30115.log.gz 2026-02-22T20:34:21.594 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71071.log 2026-02-22T20:34:21.594 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41916.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41916.log.gz 2026-02-22T20:34:21.594 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52137.log 2026-02-22T20:34:21.595 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71071.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71071.log.gz 2026-02-22T20:34:21.595 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122999.log 2026-02-22T20:34:21.596 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52137.log: 94.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86005.log.gz 2026-02-22T20:34:21.596 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52137.log.gz 2026-02-22T20:34:21.596 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127009.log 2026-02-22T20:34:21.597 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56680.log 2026-02-22T20:34:21.597 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127009.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127009.log.gz 2026-02-22T20:34:21.597 INFO:teuthology.orchestra.run.trial066.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122999.log.gz 2026-02-22T20:34:21.606 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98764.log 2026-02-22T20:34:21.607 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33005.log 2026-02-22T20:34:21.607 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56680.log: /var/log/ceph/cluster2-client.mirror.98764.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56680.log.gz 2026-02-22T20:34:21.607 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127495.log 2026-02-22T20:34:21.608 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33005.log: 87.9% 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98764.log.gz 2026-02-22T20:34:21.608 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.33005.log.gz 2026-02-22T20:34:21.608 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121371.log 2026-02-22T20:34:21.609 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.127495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83701.log 2026-02-22T20:34:21.610 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.121371.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121371.log.gz 2026-02-22T20:34:21.610 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48616.log 2026-02-22T20:34:21.611 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83701.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83701.log.gz 2026-02-22T20:34:21.611 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78479.log 2026-02-22T20:34:21.612 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48616.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48616.log.gz 2026-02-22T20:34:21.612 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83233.log 2026-02-22T20:34:21.612 INFO:teuthology.orchestra.run.trial066.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56322.log.gz 2026-02-22T20:34:21.613 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78479.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78479.log.gz 2026-02-22T20:34:21.613 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124260.log 2026-02-22T20:34:21.614 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62980.log 2026-02-22T20:34:21.614 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.83233.log: /var/log/ceph/cluster2-client.mirror.124260.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124260.log.gz 2026-02-22T20:34:21.617 INFO:teuthology.orchestra.run.trial066.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.83233.log.gz 2026-02-22T20:34:21.617 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40653.log 2026-02-22T20:34:21.618 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65173.log 2026-02-22T20:34:21.621 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.62980.log: /var/log/ceph/cluster2-client.mirror.40653.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62980.log.gz 2026-02-22T20:34:21.621 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119016.log 2026-02-22T20:34:21.622 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65173.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65173.log.gz 2026-02-22T20:34:21.622 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80447.log 2026-02-22T20:34:21.623 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119016.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119016.log.gz 2026-02-22T20:34:21.624 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48000.log 2026-02-22T20:34:21.627 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80447.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80447.log.gz 2026-02-22T20:34:21.627 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42846.log 2026-02-22T20:34:21.628 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48000.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48000.log.gz 2026-02-22T20:34:21.628 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96600.log 2026-02-22T20:34:21.629 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42846.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42846.log.gz 2026-02-22T20:34:21.629 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25309.log 2026-02-22T20:34:21.630 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96600.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96600.log.gz 2026-02-22T20:34:21.630 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62188.log 2026-02-22T20:34:21.631 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25309.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25309.log.gz 2026-02-22T20:34:21.631 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35874.log 2026-02-22T20:34:21.632 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62188.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62188.log.gz 2026-02-22T20:34:21.632 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16601.log 2026-02-22T20:34:21.633 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.35874.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35874.log.gz 2026-02-22T20:34:21.633 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125276.log 2026-02-22T20:34:21.633 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.16601.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16601.log.gz 2026-02-22T20:34:21.634 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114248.log 2026-02-22T20:34:21.634 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125276.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125276.log.gz 2026-02-22T20:34:21.635 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57983.log 2026-02-22T20:34:21.635 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.114248.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114248.log.gz 2026-02-22T20:34:21.636 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22333.log 2026-02-22T20:34:21.636 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57983.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57983.log.gz 2026-02-22T20:34:21.637 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115206.log 2026-02-22T20:34:21.637 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22333.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.22333.log.gz 2026-02-22T20:34:21.638 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113113.log 2026-02-22T20:34:21.639 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115206.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115206.log.gz 2026-02-22T20:34:21.639 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81225.log 2026-02-22T20:34:21.640 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113113.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113113.log.gz 2026-02-22T20:34:21.640 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108956.log 2026-02-22T20:34:21.641 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81225.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81225.log.gz 2026-02-22T20:34:21.641 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100064.log 2026-02-22T20:34:21.642 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108956.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108956.log.gz 2026-02-22T20:34:21.642 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53571.log 2026-02-22T20:34:21.643 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100064.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100064.log.gz 2026-02-22T20:34:21.643 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69304.log 2026-02-22T20:34:21.644 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53571.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53571.log.gz 2026-02-22T20:34:21.644 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42567.log 2026-02-22T20:34:21.645 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69304.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69304.log.gz 2026-02-22T20:34:21.645 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76414.log 2026-02-22T20:34:21.646 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42567.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42567.log.gz 2026-02-22T20:34:21.646 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125704.log 2026-02-22T20:34:21.647 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76414.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76414.log.gz 2026-02-22T20:34:21.647 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105524.log 2026-02-22T20:34:21.648 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125704.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125704.log.gz 2026-02-22T20:34:21.648 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94002.log 2026-02-22T20:34:21.654 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123354.log 2026-02-22T20:34:21.661 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94002.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88870.log.gz 2026-02-22T20:34:21.663 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94002.log.gz 2026-02-22T20:34:21.666 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87978.log 2026-02-22T20:34:21.667 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123354.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123354.log.gz 2026-02-22T20:34:21.668 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95726.log 2026-02-22T20:34:21.668 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87978.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87978.log.gz 2026-02-22T20:34:21.669 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46101.log 2026-02-22T20:34:21.669 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.95726.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.95726.log.gz 2026-02-22T20:34:21.670 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126379.log 2026-02-22T20:34:21.670 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46101.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46101.log.gz 2026-02-22T20:34:21.670 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128631.log 2026-02-22T20:34:21.671 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126379.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126379.log.gz 2026-02-22T20:34:21.671 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98248.log 2026-02-22T20:34:21.673 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128631.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128631.log.gz 2026-02-22T20:34:21.673 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49581.log 2026-02-22T20:34:21.674 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98248.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98248.log.gz 2026-02-22T20:34:21.674 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49825.log 2026-02-22T20:34:21.674 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49581.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49581.log.gz 2026-02-22T20:34:21.675 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89874.log 2026-02-22T20:34:21.675 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49825.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49825.log.gz 2026-02-22T20:34:21.676 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117423.log 2026-02-22T20:34:21.676 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89874.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89874.log.gz 2026-02-22T20:34:21.677 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37400.log 2026-02-22T20:34:21.678 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.117423.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.117423.log.gz 2026-02-22T20:34:21.678 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26297.log 2026-02-22T20:34:21.679 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37400.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37400.log.gz 2026-02-22T20:34:21.679 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127346.log 2026-02-22T20:34:21.680 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26297.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26297.log.gz 2026-02-22T20:34:21.680 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73396.log 2026-02-22T20:34:21.681 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127346.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127346.log.gz 2026-02-22T20:34:21.681 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27227.log 2026-02-22T20:34:21.682 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73396.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73396.log.gz 2026-02-22T20:34:21.682 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131208.log 2026-02-22T20:34:21.683 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27227.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27227.log.gz 2026-02-22T20:34:21.683 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37059.log 2026-02-22T20:34:21.684 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.131208.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.131208.log.gz 2026-02-22T20:34:21.684 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80162.log 2026-02-22T20:34:21.685 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37059.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37059.log.gz 2026-02-22T20:34:21.685 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78269.log 2026-02-22T20:34:21.685 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80162.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.80162.log.gz 2026-02-22T20:34:21.686 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123763.log 2026-02-22T20:34:21.686 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78269.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78269.log.gz 2026-02-22T20:34:21.687 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119619.log 2026-02-22T20:34:21.687 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123763.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123763.log.gz 2026-02-22T20:34:21.688 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91634.log 2026-02-22T20:34:21.688 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119619.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119619.log.gz 2026-02-22T20:34:21.689 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126626.log 2026-02-22T20:34:21.689 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.91634.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91634.log.gz 2026-02-22T20:34:21.690 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117130.log 2026-02-22T20:34:21.690 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126626.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126626.log.gz 2026-02-22T20:34:21.690 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56122.log 2026-02-22T20:34:21.691 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117130.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117130.log.gz 2026-02-22T20:34:21.691 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72001.log 2026-02-22T20:34:21.692 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56122.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56122.log.gz 2026-02-22T20:34:21.692 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84802.log 2026-02-22T20:34:21.693 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72001.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72001.log.gz 2026-02-22T20:34:21.693 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32110.log 2026-02-22T20:34:21.702 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68804.log 2026-02-22T20:34:21.703 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32110.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32110.log.gz 2026-02-22T20:34:21.710 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62672.log 2026-02-22T20:34:21.711 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68804.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68804.log.gz 2026-02-22T20:34:21.718 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.107480.log 2026-02-22T20:34:21.719 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62672.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62672.log.gz 2026-02-22T20:34:21.734 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73489.log 2026-02-22T20:34:21.743 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.2.107480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123092.log 2026-02-22T20:34:21.743 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73489.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73489.log.gz 2026-02-22T20:34:21.751 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48802.log 2026-02-22T20:34:21.751 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123092.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123092.log.gz 2026-02-22T20:34:21.759 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73454.log 2026-02-22T20:34:21.759 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48802.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48802.log.gz 2026-02-22T20:34:21.766 INFO:teuthology.orchestra.run.trial066.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.107480.log.gz 2026-02-22T20:34:21.771 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.19983.log 2026-02-22T20:34:21.771 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73454.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73454.log.gz 2026-02-22T20:34:21.783 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43497.log 2026-02-22T20:34:21.795 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.19983.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.19983.log.gz 2026-02-22T20:34:21.795 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72931.log 2026-02-22T20:34:21.795 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43497.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43497.log.gz 2026-02-22T20:34:21.796 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84355.log.gz 2026-02-22T20:34:21.797 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83596.log 2026-02-22T20:34:21.797 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72931.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72931.log.gz 2026-02-22T20:34:21.798 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33040.log 2026-02-22T20:34:21.799 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83596.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83596.log.gz 2026-02-22T20:34:21.800 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26890.log 2026-02-22T20:34:21.800 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33040.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33040.log.gz 2026-02-22T20:34:21.800 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109090.log 2026-02-22T20:34:21.801 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26890.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.26890.log.gz 2026-02-22T20:34:21.801 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94666.log 2026-02-22T20:34:21.815 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.109090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76834.log 2026-02-22T20:34:21.819 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94666.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40653.log.gz 2026-02-22T20:34:21.822 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28250.log 2026-02-22T20:34:21.823 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76834.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76834.log.gz 2026-02-22T20:34:21.837 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36604.log.gz 2026-02-22T20:34:21.838 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129377.log 2026-02-22T20:34:21.839 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.28250.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28250.log.gz 2026-02-22T20:34:21.840 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75074.log 2026-02-22T20:34:21.841 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.129377.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.129377.log.gz 2026-02-22T20:34:21.841 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26483.log 2026-02-22T20:34:21.841 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75074.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75074.log.gz 2026-02-22T20:34:21.841 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116283.log 2026-02-22T20:34:21.842 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26483.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26483.log.gz 2026-02-22T20:34:21.842 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40316.log 2026-02-22T20:34:21.843 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116283.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116283.log.gz 2026-02-22T20:34:21.843 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51499.log 2026-02-22T20:34:21.844 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40316.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40316.log.gz 2026-02-22T20:34:21.844 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80597.log 2026-02-22T20:34:21.845 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51499.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51499.log.gz 2026-02-22T20:34:21.845 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77009.log 2026-02-22T20:34:21.846 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.80597.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.admin.80597.log.gz 2026-02-22T20:34:21.846 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69769.log 2026-02-22T20:34:21.847 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77009.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77009.log.gz 2026-02-22T20:34:21.847 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46880.log 2026-02-22T20:34:21.848 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69769.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69769.log.gz 2026-02-22T20:34:21.848 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22021.log 2026-02-22T20:34:21.849 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46880.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46880.log.gz 2026-02-22T20:34:21.849 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105988.log 2026-02-22T20:34:21.850 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22021.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.22021.log.gz 2026-02-22T20:34:21.850 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36452.log 2026-02-22T20:34:21.850 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105988.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105988.log.gz 2026-02-22T20:34:21.851 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30487.log 2026-02-22T20:34:21.854 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36452.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36452.log.gz 2026-02-22T20:34:21.855 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24158.log 2026-02-22T20:34:21.856 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30487.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30487.log.gz 2026-02-22T20:34:21.856 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51983.log 2026-02-22T20:34:21.857 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.24158.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24158.log.gz 2026-02-22T20:34:21.857 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31048.log 2026-02-22T20:34:21.857 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.51983.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51983.log.gz 2026-02-22T20:34:21.857 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99194.log 2026-02-22T20:34:21.858 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31048.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31048.log.gz 2026-02-22T20:34:21.858 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70792.log 2026-02-22T20:34:21.859 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99194.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99194.log.gz 2026-02-22T20:34:21.859 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46973.log 2026-02-22T20:34:21.860 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70792.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70792.log.gz 2026-02-22T20:34:21.860 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53536.log 2026-02-22T20:34:21.861 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46973.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46973.log.gz 2026-02-22T20:34:21.861 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46287.log 2026-02-22T20:34:21.862 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53536.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53536.log.gz 2026-02-22T20:34:21.862 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93004.log 2026-02-22T20:34:21.863 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46287.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46287.log.gz 2026-02-22T20:34:21.863 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34679.log 2026-02-22T20:34:21.865 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93004.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93004.log.gz 2026-02-22T20:34:21.865 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103042.log 2026-02-22T20:34:21.865 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34679.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34679.log.gz 2026-02-22T20:34:21.865 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85225.log 2026-02-22T20:34:21.867 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103042.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103042.log.gz 2026-02-22T20:34:21.867 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83785.log 2026-02-22T20:34:21.867 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85225.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63172.log.gz 2026-02-22T20:34:21.867 INFO:teuthology.orchestra.run.trial066.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85225.log.gz 2026-02-22T20:34:21.867 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128726.log 2026-02-22T20:34:21.867 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130866.log 2026-02-22T20:34:21.869 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128726.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83785.log.gz 2026-02-22T20:34:21.869 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128726.log.gz 2026-02-22T20:34:21.869 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117634.log 2026-02-22T20:34:21.869 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121892.log 2026-02-22T20:34:21.869 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117634.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117634.log.gz 2026-02-22T20:34:21.869 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16498.log 2026-02-22T20:34:21.871 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121892.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121892.log.gz 2026-02-22T20:34:21.871 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121867.log 2026-02-22T20:34:21.871 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.16498.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16498.log.gz 2026-02-22T20:34:21.871 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102649.log 2026-02-22T20:34:21.872 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121867.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121867.log.gz 2026-02-22T20:34:21.872 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61909.log 2026-02-22T20:34:21.879 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102649.log: 97.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102649.log.gz 2026-02-22T20:34:21.880 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26669.log 2026-02-22T20:34:21.880 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61909.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61909.log.gz 2026-02-22T20:34:21.880 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29034.log 2026-02-22T20:34:21.881 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26669.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26669.log.gz 2026-02-22T20:34:21.881 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123305.log 2026-02-22T20:34:21.882 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29034.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29034.log.gz 2026-02-22T20:34:21.882 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25867.log 2026-02-22T20:34:21.883 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123305.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123305.log.gz 2026-02-22T20:34:21.883 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32168.log 2026-02-22T20:34:21.884 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25867.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.25867.log.gz 2026-02-22T20:34:21.884 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131893.log 2026-02-22T20:34:21.884 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32168.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32168.log.gz 2026-02-22T20:34:21.885 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26053.log 2026-02-22T20:34:21.885 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.131893.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131893.log.gz 2026-02-22T20:34:21.886 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74233.log 2026-02-22T20:34:21.886 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26053.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26053.log.gz 2026-02-22T20:34:21.886 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64371.log 2026-02-22T20:34:21.887 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74233.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74233.log.gz 2026-02-22T20:34:21.888 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75518.log 2026-02-22T20:34:21.888 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64371.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64371.log.gz 2026-02-22T20:34:21.888 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84178.log 2026-02-22T20:34:21.898 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130385.log 2026-02-22T20:34:21.914 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119436.log 2026-02-22T20:34:21.923 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.130385.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84872.log.gz 2026-02-22T20:34:21.938 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61944.log 2026-02-22T20:34:21.940 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.119436.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.119436.log.gz 2026-02-22T20:34:21.962 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30673.log 2026-02-22T20:34:21.963 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61944.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61944.log.gz 2026-02-22T20:34:21.979 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59901.log 2026-02-22T20:34:21.979 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30673.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30673.log.gz 2026-02-22T20:34:21.982 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105524.log.gz 2026-02-22T20:34:21.983 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87716.log 2026-02-22T20:34:21.983 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59901.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59901.log.gz 2026-02-22T20:34:21.984 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131001.log 2026-02-22T20:34:21.984 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.87716.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87716.log.gz 2026-02-22T20:34:21.984 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23135.log 2026-02-22T20:34:21.985 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.131001.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131001.log.gz 2026-02-22T20:34:21.985 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54838.log 2026-02-22T20:34:21.986 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23135.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.23135.log.gz 2026-02-22T20:34:21.986 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70943.log 2026-02-22T20:34:21.986 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54838.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54838.log.gz 2026-02-22T20:34:21.987 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20647.log 2026-02-22T20:34:21.987 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70943.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70943.log.gz 2026-02-22T20:34:21.987 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89807.log 2026-02-22T20:34:21.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20647.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.20647.log.gz 2026-02-22T20:34:21.988 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66696.log 2026-02-22T20:34:21.988 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.89807.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89807.log.gz 2026-02-22T20:34:21.989 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88387.log 2026-02-22T20:34:21.989 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66696.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66696.log.gz 2026-02-22T20:34:21.990 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131286.log 2026-02-22T20:34:22.003 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108308.log 2026-02-22T20:34:22.003 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.131286.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131286.log.gz 2026-02-22T20:34:22.022 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54966.log 2026-02-22T20:34:22.023 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.108308.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108308.log.gz 2026-02-22T20:34:22.035 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55768.log 2026-02-22T20:34:22.035 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54966.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54966.log.gz 2026-02-22T20:34:22.043 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49302.log 2026-02-22T20:34:22.043 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55768.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55768.log.gz 2026-02-22T20:34:22.051 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86285.log 2026-02-22T20:34:22.051 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49302.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49302.log.gz 2026-02-22T20:34:22.063 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81648.log 2026-02-22T20:34:22.077 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86285.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109090.log.gz 2026-02-22T20:34:22.091 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45113.log 2026-02-22T20:34:22.091 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.81648.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81648.log.gz 2026-02-22T20:34:22.107 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52734.log 2026-02-22T20:34:22.107 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45113.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45113.log.gz 2026-02-22T20:34:22.123 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33842.log 2026-02-22T20:34:22.123 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52734.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52734.log.gz 2026-02-22T20:34:22.138 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124209.log 2026-02-22T20:34:22.139 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33842.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33842.log.gz 2026-02-22T20:34:22.143 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88616.log.gz 2026-02-22T20:34:22.143 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113500.log 2026-02-22T20:34:22.145 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124209.log: 90.7% 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124209.log.gz 2026-02-22T20:34:22.145 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.127495.log.gz 2026-02-22T20:34:22.145 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121072.log 2026-02-22T20:34:22.146 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113500.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113500.log.gz 2026-02-22T20:34:22.146 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46508.log 2026-02-22T20:34:22.147 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.121072.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121072.log.gz 2026-02-22T20:34:22.147 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85531.log 2026-02-22T20:34:22.148 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46508.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46508.log.gz 2026-02-22T20:34:22.148 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.83031.log 2026-02-22T20:34:22.158 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46601.log 2026-02-22T20:34:22.174 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.0.83031.log: 92.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57238.log 2026-02-22T20:34:22.175 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.83031.log.gz 2026-02-22T20:34:22.175 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130866.log.gz 2026-02-22T20:34:22.175 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.46601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84678.log 2026-02-22T20:34:22.175 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46601.log.gz 2026-02-22T20:34:22.176 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57238.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57238.log.gz 2026-02-22T20:34:22.177 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83123.log 2026-02-22T20:34:22.187 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98491.log 2026-02-22T20:34:22.190 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83123.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83123.log.gz 2026-02-22T20:34:22.193 INFO:teuthology.orchestra.run.trial066.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.87307.log.gz 2026-02-22T20:34:22.202 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84553.log.gz 2026-02-22T20:34:22.202 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69455.log 2026-02-22T20:34:22.204 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75132.log 2026-02-22T20:34:22.204 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.98491.log: /var/log/ceph/cluster2-client.mirror.69455.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98491.log.gz 2026-02-22T20:34:22.204 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69455.log.gz 2026-02-22T20:34:22.204 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66231.log 2026-02-22T20:34:22.204 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54652.log 2026-02-22T20:34:22.207 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75132.log.gz/var/log/ceph/cluster2-client.mirror.66231.log: 2026-02-22T20:34:22.207 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66231.log.gz 2026-02-22T20:34:22.207 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59750.log 2026-02-22T20:34:22.207 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54652.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54652.log.gz 2026-02-22T20:34:22.207 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55152.log 2026-02-22T20:34:22.207 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94873.log 2026-02-22T20:34:22.208 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59750.log.gz/var/log/ceph/cluster2-client.mirror.55152.log: 2026-02-22T20:34:22.208 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55152.log.gz 2026-02-22T20:34:22.208 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126919.log 2026-02-22T20:34:22.208 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.94873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39444.log 2026-02-22T20:34:22.208 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94873.log.gz 2026-02-22T20:34:22.208 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126919.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126919.log.gz 2026-02-22T20:34:22.215 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125659.log 2026-02-22T20:34:22.215 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.39444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128606.log 2026-02-22T20:34:22.215 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39444.log.gz 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125659.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125659.log.gz 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36873.log 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.128606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130895.log 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.128606.log.gz 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.36873.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36873.log.gz 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110132.log 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126311.log 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.130895.log: /var/log/ceph/cluster2-client.mirror.110132.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.130895.log.gz 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110132.log.gz 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26239.log 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62749.log 2026-02-22T20:34:22.216 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126311.log: /var/log/ceph/cluster2-client.mirror.26239.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126311.log.gz 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26239.log.gz 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71036.log 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76204.log 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62749.log: /var/log/ceph/cluster2-client.mirror.71036.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62749.log.gz 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71036.log.gz 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85000.log.gz 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98105.log 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48523.log 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76204.log: /var/log/ceph/cluster2-client.mirror.98105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26983.log 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.48523.log: -- replaced with /var/log/ceph/cluster2-client.mirror.98105.log.gz 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76204.log.gz 2026-02-22T20:34:22.217 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48523.log.gz 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26611.log 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47066.log 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.26983.log: /var/log/ceph/cluster2-client.mirror.26611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77149.log 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47066.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26983.log.gz 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26611.log.gz 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47066.log.gz 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70385.log 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74570.log 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83970.log 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77149.log.gz 2026-02-22T20:34:22.218 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70385.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70385.log.gz 2026-02-22T20:34:22.220 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74570.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74570.log.gz 2026-02-22T20:34:22.220 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90904.log 2026-02-22T20:34:22.220 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125614.log 2026-02-22T20:34:22.220 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32389.log 2026-02-22T20:34:22.220 INFO:teuthology.orchestra.run.trial066.stderr: 87.9%/var/log/ceph/cluster1-client.mirror.125614.log: -- replaced with /var/log/ceph/cluster2-client.mirror.90904.log.gz 2026-02-22T20:34:22.220 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125614.log.gz 2026-02-22T20:34:22.224 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52885.log 2026-02-22T20:34:22.224 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.32389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29685.log 2026-02-22T20:34:22.224 INFO:teuthology.orchestra.run.trial066.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.52885.log: -- replaced with /var/log/ceph/cluster2-client.mirror.32389.log.gz 2026-02-22T20:34:22.224 INFO:teuthology.orchestra.run.trial066.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52885.log.gz 2026-02-22T20:34:22.224 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89967.log 2026-02-22T20:34:22.228 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.29685.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29685.log.gz 2026-02-22T20:34:22.228 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.12673.log 2026-02-22T20:34:22.228 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.89967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75854.log 2026-02-22T20:34:22.228 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.89967.log.gz 2026-02-22T20:34:22.228 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/ceph.tmp-client.admin.12673.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.12673.log.gz 2026-02-22T20:34:22.228 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119724.log 2026-02-22T20:34:22.228 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126064.log 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75854.log.gz 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119724.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119724.log.gz 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115907.log 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126064.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126064.log.gz 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37493.log 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117003.log 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115907.log.gz 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37493.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37493.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124899.log 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:22.229 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117003.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117003.log.gz 2026-02-22T20:34:22.233 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129192.log 2026-02-22T20:34:22.233 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104435.log 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124899.log.gz/var/log/ceph/cluster1-client.mirror.129192.log: 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.129192.log.gz 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47872.log 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127031.log 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.104435.log.gz 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47872.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47872.log.gz 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54559.log 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123447.log 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127031.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127031.log.gz 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.54559.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54559.log.gz 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121625.log 2026-02-22T20:34:22.234 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47438.log 2026-02-22T20:34:22.235 INFO:teuthology.orchestra.run.trial066.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123447.log.gz 2026-02-22T20:34:22.235 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121625.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.121625.log.gz 2026-02-22T20:34:22.235 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104900.log 2026-02-22T20:34:22.241 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47438.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41265.log 2026-02-22T20:34:22.241 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47438.log.gz 2026-02-22T20:34:22.241 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104900.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104900.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128786.log 2026-02-22T20:34:22.241 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:22.241 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41265.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41265.log.gz 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40800.log 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.128786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127076.log 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40800.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128786.log.gz 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40800.log.gz 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68467.log 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129213.log 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127076.log.gz 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68467.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68467.log.gz 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24902.log 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100014.log 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129213.log: /var/log/ceph/cluster2-client.mirror.24902.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129213.log.gz 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.24902.log.gz 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111926.log 2026-02-22T20:34:22.242 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115984.log 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.100014.log: /var/log/ceph/cluster2-client.mirror.111926.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100014.log.gz 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111926.log.gz 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103077.log 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63964.log 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115984.log: /var/log/ceph/cluster2-client.mirror.103077.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115984.log.gz 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103077.log.gz 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128002.log 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47252.log 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63964.log: /var/log/ceph/cluster2-client.mirror.128002.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63964.log.gz 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128002.log.gz 2026-02-22T20:34:22.243 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64429.log 2026-02-22T20:34:22.246 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57052.log 2026-02-22T20:34:22.246 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47252.log: /var/log/ceph/cluster2-client.mirror.64429.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47252.log.gz 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64429.log.gz 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28006.log 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56773.log 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57052.log: /var/log/ceph/cluster2-client.mirror.28006.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57052.log.gz 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28006.log.gz 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106612.log 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79669.log 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56773.log: /var/log/ceph/cluster2-client.mirror.106612.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56773.log.gz 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.106612.log.gz 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27634.log 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103587.log 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79669.log: /var/log/ceph/cluster2-client.mirror.27634.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79669.log.gz 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27634.log.gz 2026-02-22T20:34:22.247 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126446.log 2026-02-22T20:34:22.248 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37807.log 2026-02-22T20:34:22.248 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.103587.log: /var/log/ceph/cluster1-client.mirror.126446.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126446.log.gz 2026-02-22T20:34:22.248 INFO:teuthology.orchestra.run.trial066.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.103587.log.gz 2026-02-22T20:34:22.249 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83031.log 2026-02-22T20:34:22.249 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22056.log 2026-02-22T20:34:22.249 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37807.log: /var/log/ceph/cluster1-client.mirror.83031.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37807.log.gz 2026-02-22T20:34:22.249 INFO:teuthology.orchestra.run.trial066.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.83031.log.gz 2026-02-22T20:34:22.254 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67816.log 2026-02-22T20:34:22.255 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42102.log 2026-02-22T20:34:22.255 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22056.log: /var/log/ceph/cluster2-client.mirror.67816.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.22056.log.gz 2026-02-22T20:34:22.255 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67816.log.gz 2026-02-22T20:34:22.257 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125839.log 2026-02-22T20:34:22.257 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.42102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102515.log 2026-02-22T20:34:22.257 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42102.log.gz 2026-02-22T20:34:22.257 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125839.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125839.log.gz 2026-02-22T20:34:22.257 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59936.log 2026-02-22T20:34:22.257 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115795.log 2026-02-22T20:34:22.258 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102515.log.gz 2026-02-22T20:34:22.258 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59936.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59936.log.gz 2026-02-22T20:34:22.258 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92104.log 2026-02-22T20:34:22.258 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113535.log 2026-02-22T20:34:22.258 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115795.log.gz 2026-02-22T20:34:22.259 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.92104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75628.log 2026-02-22T20:34:22.259 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.113535.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113535.log.gz 2026-02-22T20:34:22.260 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118287.log 2026-02-22T20:34:22.260 INFO:teuthology.orchestra.run.trial066.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.92104.log.gz 2026-02-22T20:34:22.260 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.75628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111365.log 2026-02-22T20:34:22.260 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.118287.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118287.log.gz 2026-02-22T20:34:22.261 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100519.log 2026-02-22T20:34:22.261 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111365.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111365.log.gz 2026-02-22T20:34:22.268 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84802.log.gz 2026-02-22T20:34:22.270 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58727.log 2026-02-22T20:34:22.271 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94196.log 2026-02-22T20:34:22.272 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.100519.log: /var/log/ceph/cluster2-client.mirror.58727.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58727.log.gz 2026-02-22T20:34:22.272 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90188.log 2026-02-22T20:34:22.272 INFO:teuthology.orchestra.run.trial066.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.100519.log.gz 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.94196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103839.log 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75628.log.gz 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.90188.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94196.log.gz 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96227.log 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.90188.log.gz 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.103839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78409.log 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.103839.log.gz 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48244.log 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.96227.log: /var/log/ceph/cluster2-client.mirror.78409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.38851.log 2026-02-22T20:34:22.275 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.48244.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78409.log.gz 2026-02-22T20:34:22.276 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48244.log.gz 2026-02-22T20:34:22.276 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38737.log 2026-02-22T20:34:22.276 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.2.38851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49174.log 2026-02-22T20:34:22.278 INFO:teuthology.orchestra.run.trial066.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.96227.log.gz 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.38737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90504.log 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38737.log.gz 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.49174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60122.log 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49174.log.gz 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105002.log 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60122.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90504.log.gz 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.38851.log.gz 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20289.log 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60122.log.gz 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.105002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22705.log 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.20289.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105002.log.gz 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.20289.log.gz 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52102.log 2026-02-22T20:34:22.279 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33191.log 2026-02-22T20:34:22.282 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.22705.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.22705.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.20130.log 2026-02-22T20:34:22.282 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:34:22.282 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52102.log: /var/log/ceph/cluster2-client.mirror.33191.log: 88.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47496.log 2026-02-22T20:34:22.282 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52102.log.gz 2026-02-22T20:34:22.282 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.2.20130.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33191.log.gz 2026-02-22T20:34:22.282 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79354.log 2026-02-22T20:34:22.282 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131125.log 2026-02-22T20:34:22.282 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.47496.log: /var/log/ceph/cluster2-client.mirror.79354.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47496.log.gz 2026-02-22T20:34:22.283 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79354.log.gz 2026-02-22T20:34:22.283 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121963.log 2026-02-22T20:34:22.283 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93061.log 2026-02-22T20:34:22.283 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.131125.log: /var/log/ceph/cluster2-client.mirror.121963.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131125.log.gz 2026-02-22T20:34:22.283 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121963.log.gz 2026-02-22T20:34:22.284 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82291.log 2026-02-22T20:34:22.284 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23542.log 2026-02-22T20:34:22.284 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93061.log: /var/log/ceph/cluster2-client.mirror.82291.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82291.log.gz 2026-02-22T20:34:22.284 INFO:teuthology.orchestra.run.trial066.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93061.log.gz 2026-02-22T20:34:22.284 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40416.log 2026-02-22T20:34:22.285 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23542.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.23542.log.gz 2026-02-22T20:34:22.289 INFO:teuthology.orchestra.run.trial066.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.20130.log.gz 2026-02-22T20:34:22.289 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120351.log 2026-02-22T20:34:22.290 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117715.log 2026-02-22T20:34:22.290 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.120351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70757.log 2026-02-22T20:34:22.291 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.117715.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117715.log.gz 2026-02-22T20:34:22.291 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.22097.log 2026-02-22T20:34:22.292 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70757.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70757.log.gz 2026-02-22T20:34:22.292 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116639.log 2026-02-22T20:34:22.293 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.0.22097.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85786.log.gz 2026-02-22T20:34:22.293 INFO:teuthology.orchestra.run.trial066.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40416.log.gz 2026-02-22T20:34:22.293 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121419.log 2026-02-22T20:34:22.293 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91018.log 2026-02-22T20:34:22.295 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.121419.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116639.log.gz 2026-02-22T20:34:22.295 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121419.log.gz 2026-02-22T20:34:22.295 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125636.log 2026-02-22T20:34:22.296 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.91018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20664.log 2026-02-22T20:34:22.296 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125636.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91018.log.gz 2026-02-22T20:34:22.296 INFO:teuthology.orchestra.run.trial066.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125636.log.gz 2026-02-22T20:34:22.296 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23507.log 2026-02-22T20:34:22.296 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53722.log 2026-02-22T20:34:22.298 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.23507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59715.log 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.23507.log.gz 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53722.log: 88.5% 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.22097.log.gz 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64929.log 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53722.log.gz 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr: 65.5% -- replaced with /var/log/ceph/cluster2-client.mirror.20664.log.gz/var/log/ceph/cluster2-client.mirror.59715.log: 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66103.log 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59715.log.gz 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122602.log 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64929.log.gz 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88847.log 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66103.log: /var/log/ceph/cluster2-client.mirror.122602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19347.log 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66103.log.gz 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122602.log.gz 2026-02-22T20:34:22.299 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67781.log 2026-02-22T20:34:22.301 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19347.log: 84.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91433.log 2026-02-22T20:34:22.301 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19347.log.gz 2026-02-22T20:34:22.301 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.88847.log.gz 2026-02-22T20:34:22.301 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.67781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104102.log 2026-02-22T20:34:22.301 INFO:teuthology.orchestra.run.trial066.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67781.log.gz 2026-02-22T20:34:22.301 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.91433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121838.log 2026-02-22T20:34:22.301 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40188.log 2026-02-22T20:34:22.301 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104102.log: /var/log/ceph/cluster2-client.mirror.121838.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.91433.log.gz 2026-02-22T20:34:22.302 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90002.log 2026-02-22T20:34:22.302 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.40188.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121838.log.gz 2026-02-22T20:34:22.302 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40188.log.gz 2026-02-22T20:34:22.302 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72338.log 2026-02-22T20:34:22.303 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.90002.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90002.log.gz 2026-02-22T20:34:22.303 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20519.log 2026-02-22T20:34:22.303 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72338.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72338.log.gz 2026-02-22T20:34:22.303 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27785.log 2026-02-22T20:34:22.303 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.20519.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.20519.log.gz 2026-02-22T20:34:22.304 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34435.log 2026-02-22T20:34:22.304 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.27785.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27785.log.gz 2026-02-22T20:34:22.305 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93628.log 2026-02-22T20:34:22.305 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.34435.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34435.log.gz 2026-02-22T20:34:22.305 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72152.log 2026-02-22T20:34:22.306 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.93628.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.93628.log.gz 2026-02-22T20:34:22.306 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50290.log 2026-02-22T20:34:22.306 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72152.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72152.log.gz 2026-02-22T20:34:22.307 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50511.log 2026-02-22T20:34:22.307 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50290.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50290.log.gz 2026-02-22T20:34:22.308 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73919.log 2026-02-22T20:34:22.308 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.50511.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50511.log.gz 2026-02-22T20:34:22.308 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72745.log 2026-02-22T20:34:22.309 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.73919.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73919.log.gz 2026-02-22T20:34:22.309 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30022.log 2026-02-22T20:34:22.309 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.72745.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72745.log.gz 2026-02-22T20:34:22.310 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119985.log 2026-02-22T20:34:22.310 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30022.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30022.log.gz 2026-02-22T20:34:22.311 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76589.log 2026-02-22T20:34:22.311 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119985.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119985.log.gz 2026-02-22T20:34:22.311 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101709.log 2026-02-22T20:34:22.312 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76589.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76589.log.gz 2026-02-22T20:34:22.312 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127121.log 2026-02-22T20:34:22.313 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101709.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101709.log.gz 2026-02-22T20:34:22.313 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64894.log 2026-02-22T20:34:22.313 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127121.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.127121.log.gz 2026-02-22T20:34:22.314 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36576.log 2026-02-22T20:34:22.314 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64894.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64894.log.gz 2026-02-22T20:34:22.314 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56587.log 2026-02-22T20:34:22.315 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.36576.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.36576.log.gz 2026-02-22T20:34:22.315 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59099.log 2026-02-22T20:34:22.316 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56587.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56587.log.gz 2026-02-22T20:34:22.316 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33877.log 2026-02-22T20:34:22.316 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.59099.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59099.log.gz 2026-02-22T20:34:22.317 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93821.log 2026-02-22T20:34:22.317 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33877.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33877.log.gz 2026-02-22T20:34:22.318 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30243.log 2026-02-22T20:34:22.319 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.93821.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62903.log 2026-02-22T20:34:22.319 INFO:teuthology.orchestra.run.trial066.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93821.log.gz 2026-02-22T20:34:22.319 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43067.log 2026-02-22T20:34:22.320 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30243.log: /var/log/ceph/cluster2-client.mirror.62903.log: 88.3% 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62903.log.gz 2026-02-22T20:34:22.320 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.30243.log.gz 2026-02-22T20:34:22.320 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86797.log 2026-02-22T20:34:22.320 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.43067.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43067.log.gz 2026-02-22T20:34:22.321 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33749.log 2026-02-22T20:34:22.330 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58262.log 2026-02-22T20:34:22.331 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33749.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33749.log.gz 2026-02-22T20:34:22.331 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126604.log 2026-02-22T20:34:22.332 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.58262.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58262.log.gz 2026-02-22T20:34:22.332 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44706.log 2026-02-22T20:34:22.333 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126604.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126604.log.gz 2026-02-22T20:34:22.333 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88789.log 2026-02-22T20:34:22.334 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44706.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44706.log.gz 2026-02-22T20:34:22.334 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.127591.log 2026-02-22T20:34:22.338 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88789.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88387.log.gz 2026-02-22T20:34:22.339 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74419.log 2026-02-22T20:34:22.343 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.3.127591.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.127591.log.gz 2026-02-22T20:34:22.344 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19501.log 2026-02-22T20:34:22.344 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.74419.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74419.log.gz 2026-02-22T20:34:22.345 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41672.log 2026-02-22T20:34:22.345 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.admin.19501.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19501.log.gz 2026-02-22T20:34:22.346 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21821.log 2026-02-22T20:34:22.346 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41672.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41672.log.gz 2026-02-22T20:34:22.346 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118450.log 2026-02-22T20:34:22.347 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.admin.21821.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.21821.log.gz 2026-02-22T20:34:22.347 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63999.log 2026-02-22T20:34:22.348 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.118450.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118450.log.gz 2026-02-22T20:34:22.348 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83166.log 2026-02-22T20:34:22.349 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63999.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63999.log.gz 2026-02-22T20:34:22.349 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66975.log 2026-02-22T20:34:22.353 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83166.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83166.log.gz 2026-02-22T20:34:22.353 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122318.log 2026-02-22T20:34:22.353 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66975.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66975.log.gz 2026-02-22T20:34:22.354 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124522.log 2026-02-22T20:34:22.354 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.122318.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122318.log.gz 2026-02-22T20:34:22.355 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55059.log 2026-02-22T20:34:22.356 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124522.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124522.log.gz 2026-02-22T20:34:22.356 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126401.log 2026-02-22T20:34:22.357 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.55059.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55059.log.gz 2026-02-22T20:34:22.357 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30394.log 2026-02-22T20:34:22.357 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126401.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126401.log.gz 2026-02-22T20:34:22.358 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124067.log 2026-02-22T20:34:22.358 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.30394.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30394.log.gz 2026-02-22T20:34:22.359 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76309.log 2026-02-22T20:34:22.360 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.124067.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124067.log.gz 2026-02-22T20:34:22.360 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66010.log 2026-02-22T20:34:22.361 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.76309.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76309.log.gz 2026-02-22T20:34:22.361 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37152.log 2026-02-22T20:34:22.362 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.66010.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66010.log.gz 2026-02-22T20:34:22.362 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81120.log 2026-02-22T20:34:22.363 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.37152.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37152.log.gz 2026-02-22T20:34:22.363 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127324.log 2026-02-22T20:34:22.364 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.81120.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81120.log.gz 2026-02-22T20:34:22.364 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44055.log 2026-02-22T20:34:22.364 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.127324.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127324.log.gz 2026-02-22T20:34:22.365 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77744.log 2026-02-22T20:34:22.366 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.44055.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44055.log.gz 2026-02-22T20:34:22.366 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102932.log 2026-02-22T20:34:22.366 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77744.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77744.log.gz 2026-02-22T20:34:22.367 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68281.log 2026-02-22T20:34:22.367 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.102932.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.102932.log.gz 2026-02-22T20:34:22.367 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84025.log 2026-02-22T20:34:22.368 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.68281.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68281.log.gz 2026-02-22T20:34:22.368 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53385.log 2026-02-22T20:34:22.369 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.84025.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84025.log.gz 2026-02-22T20:34:22.369 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85763.log 2026-02-22T20:34:22.369 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53385.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53385.log.gz 2026-02-22T20:34:22.370 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53164.log 2026-02-22T20:34:22.370 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85763.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85763.log.gz 2026-02-22T20:34:22.371 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69269.log 2026-02-22T20:34:22.371 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.53164.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53164.log.gz 2026-02-22T20:34:22.371 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45764.log 2026-02-22T20:34:22.372 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.69269.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69269.log.gz 2026-02-22T20:34:22.372 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65022.log 2026-02-22T20:34:22.373 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45764.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45764.log.gz 2026-02-22T20:34:22.373 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126514.log 2026-02-22T20:34:22.374 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.65022.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65022.log.gz 2026-02-22T20:34:22.374 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92124.log 2026-02-22T20:34:22.375 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126514.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126514.log.gz 2026-02-22T20:34:22.375 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104400.log 2026-02-22T20:34:22.376 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.92124.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128834.log.gz 2026-02-22T20:34:22.376 INFO:teuthology.orchestra.run.trial066.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.92124.log.gz 2026-02-22T20:34:22.376 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71873.log 2026-02-22T20:34:22.377 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.104400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31703.log 2026-02-22T20:34:22.377 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.71873.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104400.log.gz 2026-02-22T20:34:22.378 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71873.log.gz 2026-02-22T20:34:22.378 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99828.log 2026-02-22T20:34:22.378 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78374.log 2026-02-22T20:34:22.378 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.99828.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31703.log.gz 2026-02-22T20:34:22.379 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99828.log.gz 2026-02-22T20:34:22.379 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96317.log 2026-02-22T20:34:22.379 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.78374.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78374.log.gz 2026-02-22T20:34:22.380 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85069.log 2026-02-22T20:34:22.380 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.96317.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41823.log 2026-02-22T20:34:22.381 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96317.log.gz 2026-02-22T20:34:22.381 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85069.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85069.log.gz 2026-02-22T20:34:22.381 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130130.log 2026-02-22T20:34:22.381 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.41823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25832.log 2026-02-22T20:34:22.382 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.130130.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41823.log.gz 2026-02-22T20:34:22.382 INFO:teuthology.orchestra.run.trial066.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130130.log.gz 2026-02-22T20:34:22.382 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116790.log 2026-02-22T20:34:22.382 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25832.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.25832.log.gz 2026-02-22T20:34:22.382 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77569.log 2026-02-22T20:34:22.383 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.116790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70420.log 2026-02-22T20:34:22.383 INFO:teuthology.orchestra.run.trial066.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116790.log.gz 2026-02-22T20:34:22.383 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.77569.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77569.log.gz 2026-02-22T20:34:22.383 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111891.log 2026-02-22T20:34:22.384 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25553.log 2026-02-22T20:34:22.384 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.111891.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70420.log.gz 2026-02-22T20:34:22.384 INFO:teuthology.orchestra.run.trial066.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111891.log.gz 2026-02-22T20:34:22.384 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.112025.log 2026-02-22T20:34:22.384 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25553.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25553.log.gz 2026-02-22T20:34:22.385 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45950.log 2026-02-22T20:34:22.385 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.0.112025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104682.log 2026-02-22T20:34:22.386 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.45950.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45950.log.gz 2026-02-22T20:34:22.386 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64708.log 2026-02-22T20:34:22.388 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.104682.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.104682.log.gz 2026-02-22T20:34:22.398 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62560.log 2026-02-22T20:34:22.399 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64708.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64708.log.gz 2026-02-22T20:34:22.399 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108333.log 2026-02-22T20:34:22.400 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62560.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62560.log.gz 2026-02-22T20:34:22.400 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88364.log 2026-02-22T20:34:22.402 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.108333.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.108333.log.gz 2026-02-22T20:34:22.402 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123234.log 2026-02-22T20:34:22.402 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88364.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88364.log.gz 2026-02-22T20:34:22.403 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57703.log 2026-02-22T20:34:22.403 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123234.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123234.log.gz 2026-02-22T20:34:22.404 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60680.log 2026-02-22T20:34:22.404 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.57703.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57703.log.gz 2026-02-22T20:34:22.404 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20030.log 2026-02-22T20:34:22.405 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.60680.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60680.log.gz 2026-02-22T20:34:22.405 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52920.log 2026-02-22T20:34:22.406 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.20030.log: 84.1% -- replaced with /var/log/ceph/cluster2-client.mirror.20030.log.gz 2026-02-22T20:34:22.406 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70664.log 2026-02-22T20:34:22.407 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.52920.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52920.log.gz 2026-02-22T20:34:22.407 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56413.log 2026-02-22T20:34:22.407 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.70664.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94666.log.gz 2026-02-22T20:34:22.408 INFO:teuthology.orchestra.run.trial066.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70664.log.gz 2026-02-22T20:34:22.408 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129544.log 2026-02-22T20:34:22.408 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.56413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110600.log 2026-02-22T20:34:22.409 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.129544.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129544.log.gz 2026-02-22T20:34:22.409 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126716.log 2026-02-22T20:34:22.411 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.110600.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110600.log.gz 2026-02-22T20:34:22.411 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20097.log 2026-02-22T20:34:22.412 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.126716.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126716.log.gz 2026-02-22T20:34:22.412 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25030.log 2026-02-22T20:34:22.413 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20097.log: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.20097.log.gz 2026-02-22T20:34:22.414 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102627.log 2026-02-22T20:34:22.414 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.25030.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25030.log.gz 2026-02-22T20:34:22.415 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85378.log 2026-02-22T20:34:22.416 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.102627.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.102627.log.gz 2026-02-22T20:34:22.416 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63813.log 2026-02-22T20:34:22.426 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.85378.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104102.log.gz 2026-02-22T20:34:22.426 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.130602.log 2026-02-22T20:34:22.427 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.63813.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63813.log.gz 2026-02-22T20:34:22.427 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101381.log 2026-02-22T20:34:22.429 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.0.130602.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.130602.log.gz 2026-02-22T20:34:22.429 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20357.log 2026-02-22T20:34:22.430 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.101381.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101381.log.gz 2026-02-22T20:34:22.430 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88327.log 2026-02-22T20:34:22.431 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.20357.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20357.log.gz 2026-02-22T20:34:22.431 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61851.log 2026-02-22T20:34:22.437 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.88327.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75518.log.gz 2026-02-22T20:34:22.438 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110726.log 2026-02-22T20:34:22.438 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61851.log: 93.7% 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61851.log.gz 2026-02-22T20:34:22.439 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.112025.log.gz 2026-02-22T20:34:22.439 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84155.log 2026-02-22T20:34:22.440 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.110726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115949.log 2026-02-22T20:34:22.440 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110726.log.gz/var/log/ceph/cluster2-client.mirror.84155.log: 2026-02-22T20:34:22.440 INFO:teuthology.orchestra.run.trial066.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84155.log.gz 2026-02-22T20:34:22.440 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52271.log 2026-02-22T20:34:22.441 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.115949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33505.log 2026-02-22T20:34:22.441 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.52271.log: -- replaced with /var/log/ceph/cluster2-client.mirror.115949.log.gz 2026-02-22T20:34:22.441 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125120.log 2026-02-22T20:34:22.441 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33505.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33505.log.gz 2026-02-22T20:34:22.442 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119759.log 2026-02-22T20:34:22.442 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.125120.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.125120.log.gz 2026-02-22T20:34:22.443 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31083.log 2026-02-22T20:34:22.443 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.119759.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119759.log.gz 2026-02-22T20:34:22.443 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33319.log 2026-02-22T20:34:22.443 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.31083.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52271.log.gz 2026-02-22T20:34:22.445 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31083.log.gz 2026-02-22T20:34:22.445 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18464.log 2026-02-22T20:34:22.445 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.33319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86938.log 2026-02-22T20:34:22.445 INFO:teuthology.orchestra.run.trial066.stderr: 88.2%/var/log/ceph/cluster2-client.admin.18464.log: -- replaced with /var/log/ceph/cluster2-client.mirror.33319.log.gz 2026-02-22T20:34:22.445 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18464.log.gz 2026-02-22T20:34:22.445 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61242.log 2026-02-22T20:34:22.446 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.86938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79214.log 2026-02-22T20:34:22.446 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.61242.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61242.log.gz 2026-02-22T20:34:22.450 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80769.log 2026-02-22T20:34:22.451 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.79214.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.79214.log.gz 2026-02-22T20:34:22.451 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123112.log 2026-02-22T20:34:22.452 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.80769.log: 85.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80769.log.gz 2026-02-22T20:34:22.452 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62502.log 2026-02-22T20:34:22.453 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.123112.log: 95.8% 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123112.log.gz 2026-02-22T20:34:22.453 INFO:teuthology.orchestra.run.trial066.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.89236.log.gz 2026-02-22T20:34:22.454 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115226.log 2026-02-22T20:34:22.454 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.62502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64743.log 2026-02-22T20:34:22.454 INFO:teuthology.orchestra.run.trial066.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62502.log.gz 2026-02-22T20:34:22.455 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster1-client.mirror.115226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83491.log 2026-02-22T20:34:22.456 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.64743.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64743.log.gz 2026-02-22T20:34:22.456 INFO:teuthology.orchestra.run.trial066.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115226.log.gz 2026-02-22T20:34:22.456 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29557.log 2026-02-22T20:34:22.456 INFO:teuthology.orchestra.run.trial066.stderr:/var/log/ceph/cluster2-client.mirror.83491.log: /var/log/ceph/cluster2-client.mirror.29557.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83491.log.gz 2026-02-22T20:34:22.456 INFO:teuthology.orchestra.run.trial066.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29557.log.gz 2026-02-22T20:34:22.583 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130385.log.gz 2026-02-22T20:34:22.652 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86285.log.gz 2026-02-22T20:34:22.686 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84178.log.gz 2026-02-22T20:34:22.740 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86938.log.gz 2026-02-22T20:34:22.762 INFO:teuthology.orchestra.run.trial066.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120351.log.gz 2026-02-22T20:34:22.767 INFO:teuthology.orchestra.run.trial066.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56413.log.gz 2026-02-22T20:34:22.811 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85531.log.gz 2026-02-22T20:34:22.837 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88327.log.gz 2026-02-22T20:34:22.883 INFO:teuthology.orchestra.run.trial066.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.88789.log.gz 2026-02-22T20:34:22.915 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85378.log.gz 2026-02-22T20:34:22.934 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86797.log.gz 2026-02-22T20:34:22.980 INFO:teuthology.orchestra.run.trial066.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83970.log.gz 2026-02-22T20:34:22.981 INFO:teuthology.orchestra.run.trial066.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.75628.log.gz 2026-02-22T20:34:23.023 INFO:teuthology.orchestra.run.trial066.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84678.log.gz 2026-02-22T20:34:23.461 INFO:teuthology.orchestra.run.trial066.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.86005.log.gz 2026-02-22T20:34:25.319 INFO:teuthology.orchestra.run.trial066.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.55836.log.gz 2026-02-22T20:34:26.467 INFO:teuthology.orchestra.run.trial066.stderr: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.19982.log.gz 2026-02-22T20:34:26.966 INFO:teuthology.orchestra.run.trial066.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.84218.log.gz 2026-02-22T20:34:27.547 INFO:teuthology.orchestra.run.trial066.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.20055.log.gz 2026-02-22T20:35:58.915 INFO:teuthology.orchestra.run.trial042.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2026-02-22T20:36:06.967 INFO:teuthology.orchestra.run.trial042.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2026-02-22T20:36:17.723 INFO:teuthology.orchestra.run.trial066.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2026-02-22T20:36:19.408 INFO:teuthology.orchestra.run.trial066.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2026-02-22T20:36:36.814 INFO:teuthology.orchestra.run.trial042.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2026-02-22T20:36:36.816 INFO:teuthology.orchestra.run.trial042.stderr: 2026-02-22T20:36:36.816 INFO:teuthology.orchestra.run.trial042.stderr:real 2m17.857s 2026-02-22T20:36:36.816 INFO:teuthology.orchestra.run.trial042.stderr:user 5m30.533s 2026-02-22T20:36:36.817 INFO:teuthology.orchestra.run.trial042.stderr:sys 0m16.013s 2026-02-22T20:37:01.277 INFO:teuthology.orchestra.run.trial066.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2026-02-22T20:37:01.278 INFO:teuthology.orchestra.run.trial066.stderr: 2026-02-22T20:37:01.278 INFO:teuthology.orchestra.run.trial066.stderr:real 2m42.308s 2026-02-22T20:37:01.278 INFO:teuthology.orchestra.run.trial066.stderr:user 7m49.700s 2026-02-22T20:37:01.278 INFO:teuthology.orchestra.run.trial066.stderr:sys 0m24.740s 2026-02-22T20:37:01.278 INFO:tasks.ceph:Archiving logs... 2026-02-22T20:37:01.278 DEBUG:teuthology.misc:Transferring archived files from trial042:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972/remote/trial042/log 2026-02-22T20:37:01.280 DEBUG:teuthology.orchestra.run.trial042:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T20:37:25.601 DEBUG:teuthology.misc:Transferring archived files from trial066:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972/remote/trial066/log 2026-02-22T20:37:25.603 DEBUG:teuthology.orchestra.run.trial066:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T20:37:58.664 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-22T20:37:58.677 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-22T20:37:58.677 DEBUG:teuthology.orchestra.run.trial042:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T20:37:58.679 DEBUG:teuthology.orchestra.run.trial066:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T20:37:59.094 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-22T20:37:59.105 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-22T20:37:59.105 DEBUG:teuthology.orchestra.run.trial042:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T20:37:59.107 DEBUG:teuthology.orchestra.run.trial066:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T20:37:59.149 INFO:teuthology.orchestra.run.trial042.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T20:37:59.149 INFO:teuthology.orchestra.run.trial042.stdout:============================================================================== 2026-02-22T20:37:59.149 INFO:teuthology.orchestra.run.trial042.stdout:+172-104-28-175. 82.35.162.146 2 u 42 128 377 3.352 +1.197 1.372 2026-02-22T20:37:59.149 INFO:teuthology.orchestra.run.trial042.stdout:+as393746.mci.tr 42.20.202.230 2 u 34 128 377 29.755 -2.312 0.639 2026-02-22T20:37:59.149 INFO:teuthology.orchestra.run.trial042.stdout:*138.89.14.60 (p .PPS. 1 u 40 128 377 5.749 -2.689 0.738 2026-02-22T20:37:59.149 INFO:teuthology.orchestra.run.trial042.stdout:-73.185.182.209 210.30.108.55 2 u 51 128 377 46.173 -7.722 5.055 2026-02-22T20:37:59.161 INFO:teuthology.orchestra.run.trial066.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T20:37:59.162 INFO:teuthology.orchestra.run.trial066.stdout:============================================================================== 2026-02-22T20:37:59.162 INFO:teuthology.orchestra.run.trial066.stdout:-172-104-28-175. 82.35.162.146 2 u 105 128 377 3.347 +9.382 1.115 2026-02-22T20:37:59.162 INFO:teuthology.orchestra.run.trial066.stdout:+nyc3.us.ntp.li 17.253.2.35 2 u 95 128 377 3.351 +3.370 3.080 2026-02-22T20:37:59.162 INFO:teuthology.orchestra.run.trial066.stdout:*138.89.14.60 (p .PPS. 1 u 40 128 377 5.926 +3.126 1.455 2026-02-22T20:37:59.162 INFO:teuthology.orchestra.run.trial066.stdout:+LAX.CALTICK.NET 17.253.83.251 2 u 26 128 377 67.586 +2.667 4.187 2026-02-22T20:37:59.163 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-22T20:37:59.182 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-22T20:37:59.182 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-22T20:37:59.204 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-22T20:37:59.224 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-22T20:37:59.243 INFO:teuthology.task.internal:Duration was 3190.053667 seconds 2026-02-22T20:37:59.244 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-22T20:37:59.263 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-22T20:37:59.263 DEBUG:teuthology.orchestra.run.trial042:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T20:37:59.266 DEBUG:teuthology.orchestra.run.trial066:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T20:37:59.293 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-22T20:37:59.294 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial042.front.sepia.ceph.com 2026-02-22T20:37:59.294 DEBUG:teuthology.orchestra.run.trial042:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T20:37:59.338 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial066.front.sepia.ceph.com 2026-02-22T20:37:59.338 DEBUG:teuthology.orchestra.run.trial066:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T20:37:59.345 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-22T20:37:59.345 DEBUG:teuthology.orchestra.run.trial042:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T20:37:59.390 DEBUG:teuthology.orchestra.run.trial066:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T20:37:59.621 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-22T20:37:59.621 DEBUG:teuthology.orchestra.run.trial042:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T20:37:59.623 DEBUG:teuthology.orchestra.run.trial066:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T20:37:59.632 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T20:37:59.632 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T20:37:59.632 INFO:teuthology.orchestra.run.trial042.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T20:37:59.632 INFO:teuthology.orchestra.run.trial042.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T20:37:59.632 INFO:teuthology.orchestra.run.trial042.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T20:37:59.634 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T20:37:59.634 INFO:teuthology.orchestra.run.trial066.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T20:37:59.634 INFO:teuthology.orchestra.run.trial066.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T20:37:59.634 INFO:teuthology.orchestra.run.trial066.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T20:37:59.634 INFO:teuthology.orchestra.run.trial066.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T20:37:59.698 INFO:teuthology.orchestra.run.trial042.stderr: 81.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T20:37:59.702 INFO:teuthology.orchestra.run.trial066.stderr: 81.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T20:37:59.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-22T20:37:59.723 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-22T20:37:59.723 DEBUG:teuthology.orchestra.run.trial042:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T20:37:59.751 DEBUG:teuthology.orchestra.run.trial066:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T20:37:59.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-22T20:37:59.773 DEBUG:teuthology.orchestra.run.trial042:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:37:59.793 DEBUG:teuthology.orchestra.run.trial066:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:37:59.798 INFO:teuthology.orchestra.run.trial042.stdout:kernel.core_pattern = core 2026-02-22T20:37:59.811 INFO:teuthology.orchestra.run.trial066.stdout:kernel.core_pattern = core 2026-02-22T20:37:59.819 DEBUG:teuthology.orchestra.run.trial042:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:37:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T20:37:59.855 DEBUG:teuthology.orchestra.run.trial066:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:37:59.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T20:37:59.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-22T20:37:59.890 INFO:teuthology.task.internal:Transferring archived files... 2026-02-22T20:37:59.892 DEBUG:teuthology.misc:Transferring archived files from trial042:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972/remote/trial042 2026-02-22T20:37:59.892 DEBUG:teuthology.orchestra.run.trial042:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T20:37:59.937 DEBUG:teuthology.misc:Transferring archived files from trial066:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63972/remote/trial066 2026-02-22T20:37:59.937 DEBUG:teuthology.orchestra.run.trial066:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T20:37:59.967 INFO:teuthology.task.internal:Removing archive directory... 2026-02-22T20:37:59.967 DEBUG:teuthology.orchestra.run.trial042:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T20:37:59.969 DEBUG:teuthology.orchestra.run.trial066:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T20:37:59.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-22T20:38:00.019 INFO:teuthology.task.internal:Not uploading archives. 2026-02-22T20:38:00.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-22T20:38:00.038 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-22T20:38:00.039 DEBUG:teuthology.orchestra.run.trial042:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T20:38:00.041 DEBUG:teuthology.orchestra.run.trial066:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T20:38:00.043 INFO:teuthology.orchestra.run.trial042.stdout: 1179687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 20:37 /home/ubuntu/cephtest 2026-02-22T20:38:00.045 INFO:teuthology.orchestra.run.trial066.stdout: 1179659 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 20:37 /home/ubuntu/cephtest 2026-02-22T20:38:00.046 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-22T20:38:00.066 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-22T20:38:00.104 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} duration: 3190.0536670684814 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-22T20:38:00.104 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T20:38:00.223 INFO:teuthology.run:pass